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ABSTRACT 

In July 1986, the Portland (Oregon) Public Schools 
began a 3-year study of the use of evaluative data in instructional 
planning and decision making in schools. This report summarizes the 
second year of the investigation, in which case studies of evaluative 
data use were conducted in one elementary school (kindergarten 
through grade 5) and one middle school (grades 6 through 8) in the 
Portland district. In the fall of 1987, the enrollment of the 
elementary school was 450 students and that of the middle school was 
550 students. The case studies collected and analyzed ethnographic 
data on the use of formal tests and informal classroom assessments. 
Interviews and surveys with 239 teachers and 35 principals provided 
information on the wide range of evaluative data sources used in 
instructional planning and decision making. Study findings provide 
the basis for a Teacher Support Guide and technical support materials 
on the effective uso of assessment information being developed in 
1988-89. More evidence of data use was found than anticipated. A key 
factor in the use of evaluative data was the principal's attitude 
about and comfort v;ith the data. Questions remain about whether the 
behaviors of principals and teachers in schools with high use of data 
are observably different from those with lower data use in schools. 
Further study of these questions will enhance the development of the 
assessment guide. Eight tables and three figures give study findings. 
Appendices A, B, and C present the case study survey, a summary of 
frequency tables, and the case study interview protocol, 
respectively. (SLD) 
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ADMINISTRATIVE SUMMARY 

THE USE OF EVALUATIVE DATA FOR 
INSTRUCTIONAL PLANNING AND DECISION MAKING 
Year Two Report: September 1987 - December 1988 



The Portland Public Schools Research and Evaluation Department study of the use 
of evaluative data in instructional planning and decision making in schools began in 1986. 
The three year research and development effort is an outgrowth of the department's 
commitment to improving the use of evaluative data in the decision making process. 
This report summarizes the second year of the investigation in which case studies of 
evaluative data use were conducted in two Portland schools. 

The research assumes that the goal of testing and assessment is to inform decision 
making. Districtwide testing and classroom assessments help educators extend successful 
basic skills instruction and identify areas where improvements may be needed. Research 
and practice show that test scores are only one aid in helping parents, teachers, and 
students make decisions about student learning. Perhaps the most important forms of 
assessment for instructional planning and decision making are those methods employed 
by teachers in the classroom. 

During 1987-88, case study research was conducted in two Portland schools, one 
elementary school and one middle school. The purpose of the case study was 1) to 
provide a comprehensive description of data use in the schools studied, 2) to provide 
validation data for the previous evaluation use studies, and 3) to test and extend the 
hypothetical decision framework and taxonomy. The case study method allowed us to 
explore the relationship between test score results and classroom assessment strategies 
to improve learning. 

The case studies collected and analyzed ethnographic data on the use of formal tests 
and informal classroom assessments. Interviews and surveys with teachers and principals 
provided information on the wide range of evaluative data sources used in instructional 
planning and decision making. Based on the findings of the study, a Teacher's 
Assessment Guide and technical support materials on the effective use of assessment 
information are being developed during 1988-89. 

Several impressions emerged from the case study which are as much a product of 
the researchers' experience as of the data. The following impressions and 
recommendations are offered in no particular order of importance: 

1. There are principals and teachers who show a commitment to assessment and are 
effective data users. These educators understand the value of the data available to 
them and make good use of it in their daily routines. We found more evidence of 
data use than anticipated, even among the high use case study schools. 



2. A key factor in whether a school is a high data use school or not is the principal's 
attitude toward and comfort with data. Principals set the tone and create the ethos 
for data use by the way they model the use of data in the instructional planning and 
decision making process. 

3. Questions remain whether the behaviors of principals and teachers in high data use 
schools are observably different from those in lower data use schools. How different 
are they? In what ways do they differ? The answer to these questions would help 
us understand how data users develop and how the central office suppo^ t system might 
better address their needs. Further study is needed on these issues. 

4, High student transiency affected, possibly in a restrictive fashion, the flexibility of data 
use among teachers in the case study schools. While the fvW effects of transiency on 
data use are not well known, we speculate that decision making is more restricted to 
the short-term, placing a greater emphasis on classroom and criterion referenced forms 
of assessment. 

5, The Research and Evaluation Department should consider developing a unified report 
of Portland Achievement Levels Tests score results, science testing, and direct writing 
assessments, 

6, The Research and Evaluation Department should develop an Assessment Guide for 
teachers and principals which might include information on (a) the role of testing and 
assessment, (b) development of effective classroom assessments, (c) preparation of 
students for testing, (d) communication with parents about assessment, and (e) use 
and interpretation of Portland Achievement Levels Tests (PALT). Training and 
dissemination of the Assessment Guide should also be implemented by the 
Department, 

7. The Research and Evaluation Department should revise the Test Coordinator's 
Manual. The manual might include information on (a) philosophy of testing in 
Portland Public Schools, (b) interpretation and use of PALT test results, (c) 
descriptions of the PALT testing program, (d) logistics of testing, (e) facsimiles of 
PALT test reports, and other information applicable to Test Coordinators. 
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INTRODUCTION AND BACKGROUND 



The Portland Public Schools Research and Evaluation Department began a multi-year, 
districtwide study of the uses of evaluative data in the instructional decision making and 
planning process in July 1986. The purpose of the study is to provide the department with 
information to guide the development of improved data collection, reporting, and 
information delivery systems. These systems would be targeted to more effectively meet the 
decision making needs of teachers, principals, parents, and administrators. 

This effort was founded on the belief that the primary purpose of evaluation is to inform 
the instructional decision making and planning process. Assessment and evaluation should 
result in better informed decisions regarding the education of the children in our schools. 
This in turn should result in improved achievement, more efficient use of school resources, 
and greater satisfaction on the part of students, parents, teachers, principals and central 
office decision makers. 

One of the basic assumptions of this project is that training teachers and principals in 
more effective and appropriate iise of achievement data will result in higher student 
achievement. However, to ascertain the training needs of teachers and administrators in the 
use of such data, we felt it necessary to conduct a thorough investigation of how data of all 
types are used in the instructional decision making and planning process in Portland schools. 

This report summarizes research for the second year of the decision making study. The 
report is intended for use by the Director of Research and Evaluation and Evaluation 
Department staff as an aid in improving the structure and operation of the testing program 
and developing improved technical support for assessment in the Portland Public Schools. 
It is expected to also be of interest to district administrators, school board members, and 
researchers interested in evaluation use and its impact on educational practice. 



CONCEPTUAL MODEL 

The study is based on a conceptual approach that combines decision making research 
with evaluation use research. Through a series of smaller studies, it seeks information to 
guide development of support services and materials that will result in improved use of 
evaluative data. The conceptual model is illustrated in Figure 1. 



Evaluative Data 

In earlier studies, we discussed evaluative data in terms of Portland Achievement Levels 
Tests (PALT). In contrast for the case study research, we expanded the definition of 
evaluative data to include: (1) P.\LT and other standardized tests, (2) teacher developed 
tests, (3) text-embedded tests, (4) performance assessments, (5) oral questioning strategies, 
(6) group assessment methods, (7) other teacher opinions, (8) classroom assignments, (9) 
student peer assessments, and (10) student self assessments. 



Figure 1, 

Conceptual Approach for Study of uao of Evaluativa Data 
in Instructional Planning and Docition Making 
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Previous Studies 



S tudy One 

A preliminary study conducted during the 1985-86 school year obtained data to guide 
the research and development process. The purpose of the preliminary study was to test 
and validate the functional utility of a hypothesized decision making structure. Figure 2.a 
shows the hypothetical decision making framework in which levels of decisions are 
associated with decision makers and their decision making time-frame. In the framework, 
decisions and the types of data associated with them can be related to the decision makers 
at each level within the school district organization. 

Figure 2.a Hypothetical Decision Framework 
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In the preliminary study, fourteen in-depth interviews were conducted with school 
principals, teachers, administrators and instructional specialists on their use of PALT test 
reports and other data. The results confirmed the usefulness of the hypothesized framework 
and revealed varied uses for evaluative data in instructional planning and decision making. 
They also led to the development of a taxonomy of decision types based on the hypothesized 
decision structure. The taxonomy is shown in Figure 2.b on the following page. 

Based on the encouraging results obtained in the preliminary study, further studies were 
planned to explore the extent of use of evaluative data, especially the PALT data, and 
examine the relationships between data use and instructional planning and decision making. 



Figure 2.b 



TAXONOMY OF DEQSION TYPBS AND EXAMPLES OF RELUVANl^ DATA 



DECISIONS 



RELEVANT DATA 



SHORT TERM 

a. Diagnose individual strengths and weaknesses 

b. Who needs remedial help 

c. What to teach (in class) 

d. What to rctcach 

e. What to emphasize 

f. What to skip 

g. How to teach 

h. How to assign grades 

i. Evaluate instructional programs 
j. Report to parents 

k. Student placement (grouping) 

I. Diagnose group strengths and weaknesses 



Standardized NRT, mdividual percentile 

Standardized NRT» individual scale scores 

Standardized NRT, individual standard scores (P) 

IRT scale score, individual (e.g. Rasch = Rrr) 

Standardized NRT, individual sub-scale score 

CRT, individual mastery score 

Standardized NRT, group percentile 

Standardized NRT» mean scale scores 

Standardized NRT, mean standard score (P) 

IRT scale score^ group mean (RTT) 

IRT scale score by goal, group mean (KIT) 

Standard score by goal-group mean (P) 

Standardized deviation score-group (DEV) 

Standardized growth (Grow) 

Standardized residua! (R) 

Attendance (frequency count) 

Behavioral observation 

Student attitude survey 

Teacher made test of skills or knowledge 

Student enrollment data 

Student grades by class 

Grade point average (individual) 

Grade point average (class or grade group) 

Graduation standard status 



MID TERM 



a. Student selection (program) 

b. Student referral (special ed.) 
c Where to allocate resources 
d. Student suspension/explusion 



e. Who receives a standard diploma 
t Who receives a modified diploma 
g. Who to advance/promote/retain 



LONG TERM 

a. What courses to offer 



b. How many sections of a course to offer 

c. How many teachers to hire 

d. What text series to adopt 

e. How to assign teachers to courses 

f. How to balance the curriculum 

g. What new programs to implement 

h. What old programs to retain or discontinue 

i. How to reduce dropouts 



A major planning phase was conducted in June and July 1986 to develop the schedules, 
refine the study questions, establish samples, and develop the interview protocols and survey 
questionnaires. This phase resulted in a two-year plan that incorporated survey research 
with a Study of the Levels of Use of evaluative data, followed by a set of case studies. 

In 1986-87, the following two-part Evaluation Use study on PALT was conducted: 

1. ) Fall 1986 - a telephone and mail survey of how parents, principals and teachers 

use Research, Evaluation and Testing Department reports and services. 

2. ) Winter 1987 - a study employing the Level of Use model (Loucks, Newlove, & 

Hall, 1975; Pechman and King, 1986), to ascertain the existing level of use of 
evaluative data in the schools. Schools were arrayed in terms of the levels of 
evaluation use revealed by the LoU model. Correlational analyses were 
conducted to examine the relationship between LoU and school achievement. 



Study Two 



Parent Teacher. Principal Surveys 



In November 1986, 273 parents of students in grades 3-8 were surveyed on their use of 
student achievement test results. Surveys were distributed at the fall Parent-Teacher 
Conferences. Results indicated that parents were pleased with the districtwide test score 
information they were receiving, but would like a report of their child's spring test scores 
and graphic, as well as numerical displays of test scores. In response, the Department 
produced a spring Parent-Teacher Conference Report which is mailed to parents. 

Also in November 1986, survey questionnaires were sent principals and teachers in a 
random sample of 40 elementary schools. This survey requested information about the use 
and usefulness of the Portland Achievement Levels Tests (PALT) reporting system. Surveys 
were completed by 274 respondents; 35 principals and 239 teachers reported on their use 
of evaluative data. 

Levels of Use Interviews 

Levels of Use (LoU) interviews were conducted in the random sample of 45 elementary 
and middle schools which did not receive the above mentioned survey. Interview subjects 
included the principal and two randomly selected teachers. In addition, interviews were 
conducted with approximately a third of the district test coordinators and all Directors of 
Instruction, Interviews were conducted during the fall of 1986 by trained, certified LoU 
inteiviewers in order to determine a districtwide baseline level of use of PALT test reports. 
The Levels of Use model identifies eight levels of use of an innovation: LoU I-nonuse, LoU 
Il-orientation, LoU Ill-preparation, LoU IVa-mechanical use, LoU IVb-routine, LoU V- 
refinement, LoU Vl-integration, and LoU Vll-renewal. One hundred twenty-six LoU 
interviews were conducted with teachers (n = 82), principals (n=36), and Directors of 
Instruction (n=8) in the district. 

We round that 32% of principals and teachers used PALT test reports in a no *nal 
manner (mechanical use) and 40% used PALT test reports in a stable, comfortable manner 
(routine use). Thirteen percent of the interviewees varied their use of PALT test reports 
to increase the impact on students (refinement). Eight percent combined their evaluation 
use with other colleagues to achieve a more collective impact on students (integration). 
Seven percent of the sample were found to be at nonuse, orientation, or preparation levels 
for using PALT test reports. Recommendations from the LOU interviews were used to 
guide improvements in the PALT reporting system. The interview transcripts also provided 
a rich source of data for further review and analysis as the case study data were interpreted. 



CASE STUDY PLAN AND METHODOLOGY 



Purpose of the Study 

A multi-site case study was conducted in 1987-88. The purpose of the case study was 
to provide a comprehensive description of the full range of data use in the schools studied, 
to provide validation data for the survey and Levels of Use studies, and to test and extend 
the hypothetical decision framework and taxonomy. The case study had four objectives: 

L To obtain an in-depth understanding of the instructional decision making process at 
the classroom and school level. 

2. To develop a comprehensive description of the planning and decision making process 
including the kinds of data used and how they are used in this process. 

3. To validate the Levels of Use findings. 

4. To test the validity of and expand the hypothetical decision framework and taxonomy. 

As originally planned, the case study would focus on two low and two high Levels of Use 
schools. One low and one high LoU school would be a K-5 elementary and one each would 
be a grade 6-8 middle school. Unfortunately, we had difficulty obtaining the cooperation 
of the principals and teachers in the low use schools. Thus, we were left with one 
elementary and one middle school, both of which were high data use schools. They were 
also both schools with low overall achievement scores as measured by districtwide tests. 
Staff scheduling problems and resource factors led us to focus more of our efforts on the 
elementary school than the middle school. Reporting of the case study was further 
hampered by one author's departure from Portland in July 1988 to take a position with the 
Colorado Springs School District. 

Case Study Plan 
Research Protocol 

A research protocol was developed to structure the case study and place boundaries 
around the data collection. The protocol included research questions, methods, instruments, 
procedures for data collection and analysis, timeline, and other relevant background 
information (Yin, 1986; Miles & Huberman, 1985). The case study was based on the 
following guidelines. 

1. Research questions should be narrow enough to provide clear guidance, but broad 
enough to not be trivial or result in too restrictive a view of the phenomenon siudied. 

2. Methods should be qualitative, varied, and redundant enough for internal validity. 



3. Instrumentation should be adequate to provide necessary structure and focus, but not 
so much as to restrict the investigation from its primary purpose, which is discovery. 



The case study included three basic types of data collection activities: intt A'iews, survey 
questionnaires, and observations in classrooms, at meetings, and other events. Where 
appropriate and useful, documents and forms used by teachers and principals for data 
collection, reporting, and interpretation were also collected. The data collection methods 
were intended to help us develop answers to the specific research questions shown below. 

Research Questions and Methods 

Research Questions Data Collection Methods 

1. How is evaluative data used for instructional planning/decision making in schools? 

a. How is data used currently by a. Principal and teacher survey, 
principals, teachers and others? interviews, classroom observations, 

staff meetings, documents. 

b. What decisions do they make using b. Validate and revise the hypothetical 
data routinely? decision framework by interaction 

with decision makers. 

c. How can the information be used for c. Interviews, observation of staff 
future planning? planning meetings. 

2. How can measurement and evaluation best serve decision making needs of principals, 
teachers, parents, and administrators? 

a. Who makes what decisions and in a. Validate and revise the hypothetical 
what timeframe? decision framework by interaction 

with decision makers. 

b. What kind of data are used? b. Observation, interviews and survey. 



The case studies examined the uses of evaluative data in instructional planmng and 
decision making in a variety of circumstances. Survey data were analyzed for both pooled 
and between school effects. As field notes were compiled, interpreted, and analyzed the 
LoU interview transcripts were re-examined to see what additional data could be derived 
that would assist in understanding the relationships between decision making and data use. 

The Case Study Schools 

One elementary school (K-5) and one middle school (6-8) were selected and agreed to 
participate in the case study. Both schools had been identified as high data use schools by 
the LoU study the previous year. The elementary school received an LoU rating of IV-B, 
indicating that it was at the "refinement" level of use of PALT data. The middle school was 
identified as being at LoU V, "integration" level of use of PALT. The Levels of Use manual 
(Loucks, Newlove, & Hall, 1975) defines these two stages in the following manner. 



Level IVB - Refinement: State in which the user varies the use of the 
innovation to increase the impact on clients within immediate sphere of 
influence. Variations are based on knowledge of both short-term and 
long-term consequences for clients. 

Level V - Integration: State in which the user is combining own efforts 
to use the innovation with related activities of colleagues to achieve a 
collective impact on clients within their common sphere of influence. 



The Elementary School 

The selected elementary school is located in a predominantly white, lower 
socioeconomic area of Portland, It reported an enrollment of 450 students in 
kindergarten through grade five during the fall of 1987, The building which was 
constructed in the early 1950's, contains 25 classrooms. The principal, a white male 
Ph.D. in his mid-fifties, was transferred to the school in the fall of 1986. His staff 
consisted of twenty-six certificated teachers, five instructional aides, two 
clerical/secretarial staff, and seven other classified employees for a total of forty 
employees. During the 1987-88 school year, the pupil-teacher ratio at the school was 
22:1, while the pupil-instructional staff ratio was 18:1. The stability index, the percentage 
of students enrolled during the opening days of school in the fall who are still enrolled 
in the same school in June, was .662; the average stability index for Portland 
elementary/middle schools in 1987-88 was .788. Special programs in which students 
participated were Chapter 1 reading (37% of the students enrolled) and math (30%), 
federal lunch (78%), low income allocation (42%), talented and gifted (4%), and special 
education (14%). The school does not have an ESL/bilingual program. Achievement 
levels in reading, language usage, and mathematics are somewhat below the district 
averages for the three grade levels tested by PALT (grades 3-5), 



The Middle School 

The selected middle school is located in a racially mixed, lower socioeconomic area 
of Portland, Enrollment in the school was 550 students in grades six through eight in 
the fall of 1987. This middle school had a staff of one assistant principal, thirty 
certificated teachers, six instructional aides, four other certificated staff, three secretarial, 
and eight other classified staff for a total of fifty-two employees. The principal, a white 
male Ph,D. in his early forties, had been principal at the school for four years. The 
building, originally constructed in 1925, v/as renovated in the early 1980's. The pupil- 
teacher ratio was 22:1, while the pupil-instructional staff ratio was 21:1. The middle 
school's stability index was .651; the average stability index for Portland 
elementary/middle schools was .788 in 1987-88. Student participation in special programs 
at the school included Chapter 1 reading (27% of the students enrolled) and math 
(27%?), federal lunch (52%), low income allocation (44%), talented and gifted (8%), 
ESL/bilingual (7%), special education (8%), and alternative programs (3%). 
Achievement levels were slightly below the district averages across grades 6-8 for reading 
and math, and right at or slightly above the district average for language at all grades. 
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CASE STUDY SURVEY 

In April 1988, survey questionnaires were administered to the principals and teachers 
in the two selected case study schools. Questionnaires were completed by 2 principals, 
24 teachers in the middle school, and 13 teachers in the elementary school. The three 
page questionnaire contained both structured response and open-ended questions (see 
Appendbc A). The survey asked questions on the type and frequency of use of data 
sources and test reports; the types of test scores used by the types of decisions made; 
type of test score groupings used; needs for other types of data and services; and 
demographic data on the respondents, e.g. level of education, number of years experience. 

Survey Results: Pooled Data 

Survey data were analyzed using SPSS Crosstabs to determine what categories of 
information were used for instructional decision making, which PALT reports were used 
and for what purposes, and how test scores were used by the teachers and principals. 

Preferred Data Sources 

Table 1 shows the pooled results for both schools on the most frequently used 
sources of data for instructional planning when no specific purpose is stated. The results 
show a strong preference for classroom assessment strategies. Performance 
assessment/teacher observation and judgment ranks first, with oral questioning strategies, 
teacher-made tests, and assignments in second, third and fourth place respectively. The 
PALT test data ranked eighth overall with 43% of the respondents indicating that they 
used it often or always. Opinions of other teachers ranked seventh, with 44% indicating 
they used it either often or always. These results extended and supported results of the 
earlier survey in which teacher-made tests ranked first among teachers as the most often 
used source of information. However in that earlier survey, performance assessment and 
oral questioning strategies were not included as response options. 

Preferred Report Types 

Survey respondents were asked to indicate the extent to which they used PALT test 
reports from two different categories: Student Achievement reports - those designed 
specifically for use by the classroom teacher, and Administrative reports - those designed 
for use by principals and central office administrators. The results for this question are 
shown in Table 2. 

The pooling of survey data from the elementary and middle schools allowed some 
interesting comparisons in the stated use of the PALT test reports. Table 2 indicates 
that the Middle School Feeder Reports (Alpha and Rank) are little used, when actually 
this is an anomaly resulting from the pooled data. Middle school feeder reports are not 
issued to elementary schools. Also of interest is the fact that several reports which are 
produced in quantities sufficient just for the principal seem to be receiving wider 
distribution to the teachers. Among these reports are the Fall-Spring Match Report, 
Preliminary and Final Principals Reports, Labels for Student Cum Folders, and Middle 
School Feeder Reports. 



Table 1. Most Frequently Used Sources of Data (pooled, N=39) 



DATA SOURCES USED IN 


Always/ 


Some- 


Seldom/ 


INSTRUCTIONAL PLANNING 


Often 


times 


Never 




("Percent of Resoondents Answering) 


Performance Assessment 


93.9 


6.1 


0.0 


Oral Questioning 


87.6 


6.3 


6.3 


Teacher-Made Tests 


74.2 


17.1 


8.6 


Class Assignments 


65.6 


15.6 


18.8 


Textbook Tests 


47.0 


29.4 


23.5 


Other Sources 


45.5 


0.0 


54.4 


Other Teachers' Opinions 


44.1 


41.2 


14.7 


PALT Tests 


42.9 


20.0 


37.2 


Group Assessments 


36.4 


48.5 


15.2 


Other Standardized Tests 


34.3 


28.6 


37.1 


Student Self Ratings 


29.1 


25.8 


45.1 


Markham Math Diagnostic 


22.6 


3.2 


74.2 


Student Peer Ratings 


12.5 


15.6 


71.9 


School Based Test Reporting 


6.6 


10.0 


83.3 


BRAG:Basal Reading Assmt, 


63 


3.1 


90.6 


Computerized Adaptive Test 


3.4 


0.0 


96.6 



Among the Student Achievement reports, the Fall-Spring Match Report was the 
most frequently used PALT test report. This is a report that includes both fall and 
spring achievement test scores for each individual student within a grade level. The 
Student Goal Report/Class Alpha was mentioned as the second most frequently used 
report. The Preliminary Principal's Report, a ranking of students in terms of their fall 
test scores was third and the Parent-Teacher Conference Report was fourth. Reports 
used the least were labels for the student cumulative folders and test score cards. Each 
of these last reports had 50% or more of the respondents indicating they were never or 
seldom used. 

Among the Administrative reports, the Grade Goal Progress Report was the most 
frequently used with 48% of teachers and principals in the case study schools indicating 
they used it often or always. Of seven administrative reports listed, this was the only one 
receiving less than 60% seldom or never responses. This pattern of use is consistent with 
the fact that these reports were designed primarily for use by administrators. 
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Table 2. Mos*. Frequently Used PALT Reports (pooled, N = 39) 



PALT REPORTS USED FOR 


Always/ 


Some- 


Seldom/ 


INSTRUCTIONAL PLANNING 


Often 


times 


Never 


Achievement Reuorts 


(Percent of 


Respondents 


Answering") 


Fall-Spring Match Report by grade 


50.0 


25.0 


25.0 


Student Goal Report/Class Alpha 


47.8 


30.4 


21.7 


Preliroinary Principals - Rank 


35.0 


30.0 


35.0 


Parent-Teacher Conference Report 


33.3 


22.2 


44.4 


Preliminary Principals - Alpha 


JU.4 


7/1 Q 
J4.0 


"XA 7 
Jt. / 


Final Principals Report - Fall 


30.0 


30.0 


40.0 


Test Score Cards 


30.0 


20.0 


50.0 


Labels for Student Cum Folders 


28.6 


28.6 


42.9 


Middle School Alpha Feeder 


23.1 


0.0 


76.9 


Middle School Rank Feeder 


20.0 


20.0 


60.0 


Administrative Reoorts 


CPercent of 


Respondents 


Answering) 


Grade Goal Progress Report 


47.6 


19.0 


33.3 


Achievement Profiles 


27.8 


11.1 


61.1 


Five Year Achievement Gains Rpt. 


21.1 


15.8 


63.2 


Five Year Growth Charts 


18.8 


18.8 


62.6 


Achievement Rpt. by Ethnic Group 


12.5 


6.3 


81.3 


Board Rpt. Achievement Goal #1 


6.7 


6.7 


86.7 


Board Rpt- Achievement Goal #2 


6.7 


6.7 


86.7 



Self Ratings as Data Users 

Respondents were also asked to indicate on a five point rating scale how they rated 
themselves as users of the full range of evaluative data in three situations: planning 
instruction/making decisions, developing classroom/building academic goals, and sharing 
data with students to improve achievement. The results are shown in Table 3. 

Table 3. Self Rating as Evaluative Data User by Situation (pooled, N = 39) 



SITUATIONAL USES 


Always/ 


Some- 


Seldom/ 


OF DATA 


Often 


times 


Never 




fPercentof Respondents Answering") 


Plan or Make Decisions 


57.1 


28.6 


14.3 


Develop Class/Building Goals 


55.8 


26.5 


17.6 


Share Data with Students 








to Improve Achievement 


54.3 


31.4 


14.3 
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The responses to this question are approximately the same for each of the three 
data uses specified. Respondents tended to indicate moderate to heavy use of data more 
than light or moderate use, suggesting that they viewed themselves as frequent data users. 

Preferred Test Report Formats 

Respondents were asked to check the two PALT test report formats they found most 
useful out of a list of seven. These results are shown in Table 4. 

The table shows that 64% of the respondents had a strong preference for report 
formats featuring individual student data. Reporting by instructional groups ranked 
second, with class and grade groups next. Only the principals found the school level 
formats useful and no respondents checked either cluster or district level report formats. 

When asked "do the PALT reports you receive meet all of your instructional 
decision making needs," 57% indicated "no" and 43% "yes". While it is obvious that 
PALT reports were not designed to meet all of a teacher's decision making needs, 43% 
is a surprisingly large "yes" response and raises some doubt regarding the interpretation 
of this item. 



Table 4. Most Useful PALT Test Report Format 
(pooled, N=39) 



REPORT FORMAT 


Percent Preferring 
Report Format 


Individual Student 


63.9 


Instructional Group 


36.1 


Class 


27.8 


Grade 


22.2 


School 


2.8 


Cluster/District 


0.0 



Survey Results: by School 

Separate analyses were conducted for each case study school to see if the results 
would follow patterns expected for an elementary and middle school. These data are 
displayed in Tables 5 through 8. 

Preferred Data Sources 

Table 5 displays the most frequently used sources of evaluative data. The preferred 
sources of data varied between the two schools. The elementary school teachers reported 
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Table 5. Most Frequently Used Data Sources by School 
(Elementary N=14, Middle School N=25) 



ELEMENTARY MIDDLE SCHOOL 

SOURCES Always/ Seldom/ Always/ Seldom/ 

OF DATA Often Some Never Often Some Never 



(Percent of Respondents Answering) 



Performance Assessment 


91.0 


9.1 


0.0 


95.4 


4.5 


0.0 


Oral Questioning 


100.0 


0.0 


0.0 


82.6 


8.7 


8.7 


Teacher-Made Tests 


50.0 


33.3 


16.7 


87.0 


8.7 


4.3 


Class Assigimients 


50.0 


30.0 


20.0 


72.7 


9.1 


18.1 


Textbook Tests 


63.6 


9.1 


27.3 


39.1 


39.1 


21.7 


Other Sources 


50.0 


0.0 


50.0 


42.9 


0.0 


57.1 


Other Teachers' Opinions 


45.5 


36.4 


18.2 


43.4 


43.5 


13.0 


PALT Tests 


58.4 


8.3 


33.4 


34.7 


26.1 


39.1 


Group Assessments 


36.4 


54.5 


9.1 


36.4 


45.5 


18.2 


Other Standardized Tests 


33.3 


25.0 


41.6 


34.8 


30.4 


34.7 


Student Self Rating 


18.2 


27.3 


54.6 


35.0 


25.0 


40.0 


Student Peer Rating 


9.1 


18.2 


72.8 


14.3 


14.3 


71.4 


Markham Math Diagnostic 


100.0 


0.0 


0.0 


33.3 


4.8 


61.9 


Computerized Adaptive Testing 


100.0 


0.0 


0.0 


5.3 


0.0 


94.7 


School Based Test Reporting 


100.0 


0.0 


0.0 


10.0 


15.0 


75.0 


BRAG:Basal Reading Assmt. 


9.1 


90.9 


0.0 


9.5 


0.0 


90.5 
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Table 6. Most Frequently Used PALT Reports by School 
(Elementary N = 14, Middle School N=25) 



ACfflEVEMENT REPORTS 



ELEMENTARY MIDDLE SCHOOL 

Always/ Seldom/ Always/ Seldom/ 
Often Some Never Often Some Never 

(Percent of Respondents Answering) 



Fall-Spring Match Report by Grade 


62.5 


12.5 


25.0 


43.8 


31.3 


25.1 


Student Goal Report/Class Alpha 


50.0 


10.0 


40.0 


46.1 


46.2 


7.7 


Preliminary Principals-Rank 


50.0 


0.0 


50.0 


25.0 


50.0 


25.0 


Parent-Teacher Conference Report 


33.3 


11.1 


55.5 


33.3 


27.8 


38.9 


Preliminary Principals-Alpha 


40.0 


30.0 


30.0 


23.1 


38.5 


38.5 


Final Principals Report-Fall 


50.0 


12.5 


37.5 


16.6 


41.7 


41.6 


Test Score Cards 


22.2 


11.1 


66.6 


36.4 


27.3 


36.4 


Labels for Student Cum Folders 


42.8 


42.9 


14.3 


21.4 


21.4 


57.1 


Middle School Alpha Feeder 


0.0 


0.0 


100.0 


30.0 


0.0 


70.0 


Middle School Rank Feeder 


0.0 


0.0 


100.0 


25.0 


25.0 


50.0 


ADMINISTRATIVE REPORTS 


ELEMENTARY 
Always/ Seldom/ 
Often Some Never 


MIDDLE SCHOOL 
Always/ Seldom/ 
Often Some Never 



(Percent of Respondents Answering) 



Grade Goal Progress Report 


55.5 


22.2 


22.2 


41.7 


16.7 


16.7 


Achievement Profiles 


28.6 


14.3 


57.1 - 


27.3 


9.1 


63.7 


Five Year Achievement Gains Rpt. 


37.5 


12.5 


50.0 


9.1 


18.2 


72.7 


Five Year Growth Charts 


33.3 


16.7 


50.0 


10.0 


20.0 


70.0 


Achievement Rpt. by Ethnic Group 


16.7 


0.0 


83.3 


20.0 


0.0 


80.0 


Board Rpt. Achievement Goal #1 


20.0 


80.0 


0.0 


10.0 


0.0 


90.0 


Board Rpt. Achievement Goal #2 


20.0 


80.0 


0.0 


10.0 


0.0 


90.0 
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a heavier reliance on PALT, textbook tests, and oral questioning strategies in the 
classroom than the middle school teachers. The middle school staff relied more heavily 
on teacher-made tests, class assignments, student self-ratings, and locally developed 
diagnostic materials than did the elementary staff. These patterns are consistent with 
expectations for schools at these respective levels in Portland given the types of data and 
materials that are made available at each level. 

Preferred Report Types 

Table 6 shows the types of PALT test reports preferred by the case study schools. 
Major differences were apparent in the types of test reports used by the staff of the two 
schools. In the student achievement reports, the elementary staff reported greater use 
of the three fall principal's reports (preliminary alpha, rank, and final), the fall-spring 
match report, and the cum folder labels than the middle school. The middle school staff 
reported more frequent use of the student goal report, the test score cards, and the 
parent-teacher conference report. In the administrative report package, the Grade Goal 
Progress Report and the Five Year Achievement Gains Report were more frequently 
used by the elementary than by the middle school teachers. 

Self Ratings as Data Users 

There were distinct differences between the schools on their self ratings as data 
users in different decision making situations. The elementary staff indicated a much 
higher frequency of use of data than the middle school. The elementary staff reported 
they used data more often for planning decisions and developing class or building goals. 
The middle school staff reported shared data with students to help improve achievement. 
These results are displayed in Table 7. 

Table 7. Self Rating as Data User by Situation for Each School 
(Elementary N = 14, Middle School N=25) 



USES OF DATA 


ELEMENTARY 
Always/ Seldom/ 
Often Some Never 


MIDDLE SCHOOL 
Always/ Seldom/ 
Often Some Never 




("Percent of Respondents Answering 


Plan or Make Decisions 


81.8 18.2 0.0 


45.9 33.3 20.9 


Develop Class/Bldg Goals 


81.8 18.2 0.0 


43.5 30.4 26.1 


Share Data with Students 
to Improve Achievement 


63.6 27.3 9.1 


50.0 33.3 16.7 
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INTERVIEW AND OBSERVATION RESULTS 



Structured interviews were conducted with the principals, test coordinators (who were 
also classroom teachers), and four teachers in each of the case study schools. Teachers were 
selected to establish a balance by sex and ethnicity and to represent all grades. Thus for 
each school, at least one minority teacher was included among the two male and two female 
teachers interviewed. The semi-structured interviews were conducted in the late fall and 
early winter of 1987-88 in the elementary school and in the spring and fall of 1988 in the 
middle school. In each case, the principal was the first to be interviewed. Initial interviews 
lasted approximately one hour with the principal and were somewhat shorter for teachers. 
Follow-up interviews were conducted with some participants to gain additional information 
about questions and issues as a need arose. 

In addition to these interviews, the teachers from each group agreed to be observed for 
up to four days each, over the course of the year. At the elementary school a total of 13 
hours of classroom observations were conducted with four teachers; the teachers were 
observed approximately three to five hours each. Observation time at the middle school was 
somewhat more limited, due to time constraints imposed by beginning the observations later 
in the school year. At the middle school, eight hours of observation were conducted with 
four teachers or approximately one to four hours each. 

The final data collection strategy consisted of attending staff meetings at each school. 
Five staff meetings were observed at each site. At the elementary school, three staff 
meetings were observed in November 1987, January and May 1988; at the middle school two 
staff meetings were observed in November 1987 and May 1988. 

One problem hampered the interpretation of the data. As mentioned earlier, one 
author left Portland to take a position with the Colorado Springs Public Schools in August 
1988. In the transition some field notes from the middle school were misplaced and have 
not been recovered. Therefore, results for the middle school are based in part on one 
researcher's field notes and in part on memory. Actual middle school observation and 
interview data available for analysis and interpretation include data from two principal 
interviews, four classroom observations, two staff meetings, two team meetings of sixth grade 
teachers, and four teacher interviews. 

The case study results are discussed below in two ways: 1) in terms of the role of the 
data user, and 2) in terms of the hypothetical decision framework in which short, medium 
and long term planning horizons provide the basis for data use. Since the focus of the case 
study was on the school building, the roles of the principal, test coordinator and teachers 
were examined for their use of data. 

The Principal's Role 

Both principals in this study had been identified through the LoU study as relatively 
high level users of the PALT data. Little was known about their use of other types of data 
however. Therefore, the initial case study interviews explored their knowledge and use of 
other types of data, as well as the PALT data, and pursued specific uses with them. 
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The principals proved to be quite dependent on the data provided by the PALT 
reporting system. Interviews produced noteworthy differences between the principals in 
their data use based on 1) their own background, 2) the levei of their school, and 3) the 
availability of enhancements to PALT in their school. Both principals used PALT test 
data to develop a broad view of their school's strengths and weaknesses and to help 
teachers target instruction on goal areas where the school was not as strong as it could 
be. Both principals set time aside in staff meetings several times a year to discuss test 
results and ways that they might be used to improve student performance. They also 
demonstrated detailed knowledge of the testing program, reporting system and 
appropriate uses of the metrics and reports produced. They conducted inservice with 
teachers on specific ways to interpret and use achievement data in planning instruction. 

The principals differed in the specific uses they made of data and in the level of 
detail they pursued in analyzing and applying data in their schools. The elementary 
principal, a Ph.D. with 19 years experience as a principal, had just begun his tenure in 
his school in 1986 when the LoU study was conducted. The middle school principal, a 
Ph.D. with 11 years experience as a principal, had four years experience at his school 
when the LoU study was conducted. 

The Elementary Principal 

Several uses of data idenrified by the elementary principal include: 1) school level 
decision making, 2) teacher motivation, 3) reviewing and reporting individual student 
data, and 4) decisions about resource allocation. At the school level, he correctly 
identified PALT data as found in the administrative reports as being most applicable. 
In the LoU interview, he referred to the use of the PPS Achievement Profiles report to 
compare his school to others in terms of neighborhood characteristics, stability, and other 
factors. Textbook tests were given their own level, as if they were unique sources of 
data. At the classroom level, he identified teacher-made tests as a prime example. 

School Level Use 

The elementary principal began applying data to the management of the school 
immediately upon his arrival by using the Achievement Profiles and administrative reports 
to gain a historical perspective on the school and to establish a baseline against which 
he could set goals for improvement. An example of this broad, school level perspective 
is found in the following 'excerpt from an interview with the principal. 

.„/ use the administrative reports and graphs to show generally whether the 
school is making any progress. If I see some indication that we're stagnant 
in some area or if I see repeated stagnation in a particular grade level, I 
try to see whether there was a substitute teacher that was there for the rest 
of that year or whether there was a new program brought in that year. For 
instancey in this schooL.they added a continuous progress program. I 
checked back two or three years to see whether in fact, when they added 
the new continuous progress program.. .the program didn't show any grcMh. 
It might have shown a reversal In fact, that's what I'm seeing...that as a 
schooly it may have reversed itself in terms of overall achievement. 
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The elementary principal also used the Achievement Profile data in his day to day 
counseling with parents regarding the school and its relationship to other schools with 
similar neighborhood characteristics. This counseling also used individual student test 
scores and performance histories to set expectation levels and solicit parental assistance. 

Another example of this school level focus, so appropriate for a principal, is found in 
what the principal referred to as identifying "patterns of default" in the school's goal 
achievement. The principal described using building reports to kindergarten, first and 
second grade teachers (grades at which standardized tests are not routinely given), to 
emphasize goal areas that were displaying patterns of weakness at later grades. His 
observation was that very slight weaknesses observed at lower grades would tend to increase 
over time, manifesting themselves in significant deficiencies as students progressed to higher 
grades. To identify such weaknesses he would break the test results down by goal and sub- 
goal at each grade level and track backwards for any deficiency he found at a higher grade. 
By presenting such disaggregated data to primary grade teachers he felt that he could 
motivate them to focus more instruction on the weak areas. 

This principal tended to focus more on formal examples of measurement, such as the 
PALT and other standardized testing than on informal assessments. Perhaps this is what 
he thought we wanted to know about, or perhaps it simply reflected his own orientation as 
a principal with many years of experience in the system. He displayed a good grasp of the 
value of achievement data in focusing and targeting instruction. This valuing of data in 
decision making was also relayed to teachers at the school level, 

I've capitalized on the strengths I've seen, emphasizing to the teachers that they 
spend the time and effort to identify specific teachings and goals areas where there 
are some shortcomings, and then spend some additional time - not just time in 
reading and math, teaching in those areas. 

Teacher Motivation 

Teacher motivation was another major emphasis for the elementary principal. He used 
data in a variety of ways to stimulate, interest, and motivate teachers to improve student 
performance. Most of his efforts seemed to be well received by the teachers. There were 
exceptions, however. For example, he spoke of the time he brought in a consultant from 
the Evaluation Department to explain the PALT system and the use of test scores in the 
classroom. By his own admission, many of his teachers resented the presentation, indicating 
that they already knew the system and its uses. In many informal asides with the researcher, 
the principal made it known that he felt he needed to use the data to motivate teachers to 
do a better job because his school's test scores were lower than he thought they should be. 

Specific examples of how the elementary principal used data in his efforts to motivate 
teachers are 1) presentations on the achievement profiles in staff meetings, 2) sharing 
classroom level data, such as a goal achievement by class report with an individual teacher 
to help idemify areas that need greater emphasis, 3) focused discussions in staff meetings, 
including staff meetings totally dedicated to reviewing data and making at least tentative 
decisions about areas to improve. 
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/ like to loo, at reports. I have had no problems in motivating teachers after 
I explain how to use them and what effect they can have, I have no problem. 
I have teachers jump on the bandwagon and say, "Sounds great. Let's go. " 

The principal's emphasis on teacher motivation was also evident in the numerous testing 
and achievement topics included in staff meeting agendas. One February staff meeting 
included "an informal chat about testing." This item turned out to be a carefully 
orchestrated 11/2 hour seminar on testing and assessment that included presentations by 
selected staff on test-taking skills, the use of assessment in teaching, the use of the Parent- 
Teacher Guide to the testing system, practice items and goals for the language usage test, 
the use of the Scantron for scoring classroom tests to give students realistic testing practice, 
and the use of performance assessment techniques in whole language instruction. The 
meeting agenda could have served as a syllabus for an introductory graduate course on 
applied testing and measurement. The level of participation in the discussion was high and 
teachers were enthusiastic. The distinction between teaching to the test and teaching the 
curriculum on which the test was based was made clear. By initiating this session and 
having teachers themselves make the presentations, the principal was developing teacher 
commitment Lo using the testing system in a constructive way. He was focusing them on 
preparing the students to do their best on the tests to be given two months later and 
preparing the teachers to think positively about the value of the data generated by the tests. 

Individual Student Data 

This principal made good use of achievement data to track individual student progress, 
to diagnose areas needing remediation, and to group students for instruction. For example, 
he described his use of PALT achievement reports in the following excerpt. 

We're using mem to review how a particular kid stands on progress, especially 
if he is identified for building screening committee or other problems or 
concerns. We refer to that (data) as a base point for analysis of a problem. 

In the classroom, he advocated appropriate use of test scores for individual students. 

We use the KIT scores and P-scores related to the individual results that 
we have for kids. Some of those results come in class sheets, class forms, and 
we have individual reports that we get for each kid. But when we're looking 
at teaching goals in a classroom, they would look, more precisely, to the 
individual goal reports that kids have. 

Resource Allocation Decisions 

Within the school, the elementary principal saw some value in using administrative 
report data to base decisions about resource allocations. Referring again to patterns in the 
data, he said he analyzed the results to see whether or not he needed to put in additional 
"boosters" in the form of people who are resources to a program. A booster might come 
in the form of assigning a teacher to work in a helping role as a consultant to another in 
order to achieve a greater impact fi a program. 
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Summary of the Elementary Principars Use of Data 

The elementary principal proved to be a sophisticated data user. Case study interviews 
and staff meeting observations validated the earlier LoU data that identified him. as a high 
level PALT data user. He did not demonstrate as high a level of familiarity with the use 
of non-PALT test data however. His major uses of data are summarized as follows: 

0 motivating teachers to use data for decision making 

0 identifying school wide areas of weakness to target for improvement - small 
problems at beginning grades get larger as they progress to higher grade levels 

0 tracking individual student progress - by goal area to communicate with parents 
about individual students 

0 checking placement decisions after they were made 

0 making resource allocation decisions 

In terms of the hypothetical decision framework, this principal used PALT data for 
decisions on all three time horizons. Examples of his short-term uses include talking with 
parents about their student's achievement, student referral, grouping for instruction, 
regrouping students based on test scores, and motivating teachers data use. 

This principal's mid-term uses included areas of the curriculum to emphasize, allocation 
of teacher resources, and grouping for instruction within a continuous progress program. 

The long term uses were one of this principal's strengths. He reviewed historical 
achievement data on the school to see if there were areas that had suffered from neglect 
and used the information to refocus the instructional emphasis. He used data disaggregated 
by goal area and grade level to backtrack across grade levels in an attempt to identify 
persistent problems and direct teacher attention toward them. 

Overall, this principal's uses of PALT data were consistent with his role as instructional 
leader in the school. He maintained a focus on the big picture, emphasized instruction in 
areas where he detected weaknesses, and used the PALT data as a navigational tool to help 
guide the course of the school. He also used PALT data as a means of motivating teachers 
by identifying instructional areas in need of improvement and demonstrating for them ways 
they might increase student performance in those areas. As a result of the principal s 
emphasis on PALT data use, teachers were much more conscious of the potential value of 
data and the ways in which it could be used to improve instruction. 

The Middle School Principal 

The principal of the middle school had been identified in the Levels of Use study as a 
LoU V (Integration) user of PALT test results. He has a reputation in the district as a 
data-oriented, innovative principal who takes advantage of technological innovations 
available to him. For example, his school was one 14 schools piloting a School-Based Test 
Reporting (SBTR) system, which he helped design. Through this system, each school could 
download a school file from the mainframe computer in the administration building to their 
local site using an MS-DOS PC as a terminal. The downloaded file could be manipulated 
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on-site to create customized reports for specific classes, grades, or other configurations. 
This principal was also the chairperson of the SBTR user's group. He had volunteered to 
be one of the first schools in the district to pilot the Computerized Adaptive Testing (CAT) 
system. The CAT system provides on-line, short, curriculum-based tests that yield very 
precise estimates of a student's knowledge of the basic skill area tested. 

In case study interviews, the middle school principal discussed several categories and 
factors related to his data use including: 1) school level decision making, 2) teacher 
motivation in use of test and assessment data, 3) self perceptions, 4) factors associated with 
successful data use, and 5) key points for teachers regarding the use of test and assessment 
data in instructional planning and decision making. 

School Level Use 

In his initial LoU interview in November 1985, the middle school principal discussed his 
use of test data in three time periods: 1) long term program assessment and forecasting in 
the spring, 2) mid-term intake and planning in the fall, and 3) short term, ongoing 
instructional monitoring during the school year. Tnese time periods correspond well to the 
planning horizons of the hypothetical decision framework. He described the decision 
making process and the uses of data from various sources for each time period. 

1) Spring - program assessment and forecasting. Each spring this principal would 
review PALT/SBTR test data for two major purposes: program assessment and forecasting. 
In program assessment, he would use the RIT score means presented in the Grade-Goal 
Progress Report^ to assess the overall strength of his instructional program hy grade. He 
also used the standard P-score data on the Fall-Spring Match Report to determine how 
many sections of pre-algebra and algebra he would need to plan for in the fall, how many 
Chapter 1 classes he would have, and so forth. He also made use of optional, locally 
developed, diagnostic and prognostic measures such as the Portland Prognostic Math test 
and the Markham Math diagnostic test. During the summer, he used all the data to plan 
changes in instructional emphases for the next year. When teachers returned in August, he 
inserviced them using the spring test data and they would develop their plans for the year. 

2) Fall - intake, placement, and instructional planning. In the 1985 LoU interview, the 
principal discussed his focus on longer range instructional planning with teachers in the fall. 
Incoming students with test scores were be placed in classes according to test score category 
(low, average or high). Supplementary classroom assessment materials were used in 
combination with PALT tests to create a composite picture of a student's performance. 
Teachers would then use the data to create appropriate instructional groups. Two years 
later in the 1987 case study interview, the middle school principal indicated that he had 
moved away from homogeneous grouping based on research findings and was grouping 
students in two broad groups, low to average and average to high, in terms of test results. 

Specific placement decisions, such as Chapter 1, were based on standard score cutoff 
points (P = 43). In cases where a student was placed using a marginal score (P = 42), he 
would review other sources of data such as classroom performance and grades to determine 
whether placement was appropriate. 
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3) School year instructional monitoring and short term planning. During the course of 
the year the middle school principal encouraged teachers to make use of classroom 
diagnostic tools and the CAT program to monitor student performance and guide their 
instructional processes. 

Because of his involvement with the SBTR system, this principal made strong use of the 
system's ability to retrieve a four point in time test history for a student and to summarize 
goal and class data by those four points in time. He valued the RIT score as a growth 
measure and clearly understood its limitations in other situations. 

Other Data Used 

The middle school principal used a variety of data sources for decision making, but 
relied primarily on PALT/SBTR data for building level decisions. He did not personally 
review results of the diagnostic math tests, but encouraged his teachers to use them. He 
worked with an instructional materials and software vendor to cross reference the locally 
developed Basal Reading Assessment Guide (BRAG) system to the company's materials 
so that BRAG objectives could be addressed by the commercial materials. BRAG tests 
were tb"s used to obtain prescriptions for remediation. 

He routinely reviewed behavioral data on student disciplinary and detention referrals 
from an Apple microcomputer da .abase provided by central administration. Periodically, 
he would review the attendance data checking for overall patterns. He would also review 
the eighth grade credit by exam results to check the student's achievement status against 
placement decisions. 

For individual conferences with parents, he would print out student data from SBTR to 
obtain data on the student's performance over four points in a two year time period. This 
helped a parent understand whether or not the student was progressing at a normal rate. 

Self Perception as a Data User 

The middle school principal revealed great confidence in himself as a data user and 
held a realistic view of his strengths and weaknesses. 

/ know statistical information very well. I have a thorough understanding of 
PALT I have some weaknesses in terms of special education evaluation 
criteria and I need j xmiliarity with special education diagnostic and placement 
instruments. I do well in helping teachers determine the strengths and 
weaknesses of students. 

He characterized his teachers as knowledgeable data users. 

Factors Associated with Successful Data Use 

This principal felt that the degree of trust between the principal and staff was a critical 
factor in data use by teachers. On use of the testing system he had this to say: 
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Teachers need to understand that you want to help them to help their kids 
and improve instruction. They need a better understanding of the (testing) 

# system and what it is. They need to know what is in the blue book 

(Parent-Teacher Guide to PALT) and what sub-goals are. They need to 
know what the PALT levels mean and who iiey are for. 

He felt that teacher defensiveness about data needed to be overcome and that it 
^ was the principal's role to see that this happened. 

It's important for a teacher not to be defensive about data. Growth is easy 
to explain to parents, but explaining regression is difficult. Teachers need 
help here so they can be secure in explaining why scores may have declined. 

• This concern for helping teachers to understand the testing system, its uses and the 
properties and use of the various scores it produces was evident in his presentation of 
data in staff meetings. One of the staff meetings we observed took place in November 
following the release of the fall PALT test results. In the meeting the principal 
presented the downloaded School-Based Test Report data on math and language usage 
to teachers. The presentation was a thorough and detailed explanation of the purpose, 

* philosophy, features and characteristics of the PALT and explicit instruction on how to 
use certain reports. The principal presented each teacher with a class-goal summary for 
their class, then made a presentation using overhead transparencies. He carefully 
explained what the report contained, noting that there were a number of new teachers 
in the school who might not have seen the report. 

• 

Key Points for Teachers 

The principal explained three key points in his inservice (a) the types of test scores 
used, (b) how to use the report, and (c) the overall value of the reporting system. 

* The data he used were from the fall administration of the PALT but included three 
additional data points, providing a two year history of student achievement expressed in 
both RIT and P-scores. Both student level and class summary statistics were included. 
He provided a simple, clear and concise explanation of the RIT score, a concept that 
many find difficult to grasp. He had an excellent sense of just how much detail to 

# provide to give the teachers a sense of what the score means in terms of the curriculum 
continuum " how it could best be used. He was equally expert in describing the P- 
score and its uses. 

By tracing an example of one student over time, he illustrated for the teachers the 
^ difference between "normal growth" (the same P-score from year to year) and "greater 

than normal." He demonstrated how the report could be used to decide which goal 
areas to emphasize by examining the number of students in each range (low, average, 
high) on each goal. He focused the presentation on instructional decision making by 
posing two questions: 1) "what is it that we are measuring?" answering "progress on the 
goals of the district" and 2) "how is it measured?" which he answered by introducing the 

• Parent-Teacher Guide to PALT as the source of information on goals and sub-goals. 
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The principal summarized his presentation by pointing out the value of the system in 
(a) lesson planning, (b) communicating with parents, and (c) working with individual 
students. The presentation engendered a lively question and answer session that revealed 
a high level of interest on the part of teachers. 

Summary of Middle School Principars Use of Data 

This middle school principal lived up to his reputation as a data based decision maker, 
technological innovator, and instructional leader. His level of sophistication in the use of 
formal testi^^g data was superior to that of the elementary principal. However, he was also 
more dependent on PALT data than his elementary counterpart. At the building level, he 
made less use of other types of data available to him, although he made such data available 
to teachers and encouraged their use of it. 

The principal provided more structured inservice and work group decision sessions using 
data than did the elementary principal, reflecting both a difference in their styles and a 
difference in the type and quality of data available to each. Through the SBTR system, the 
middle school principal was able to obtain a variety of reports, including historical data, not 
available to the elementary principal, thus it would be expected that the middle school 
would be more dependent on such data. Moreover, testing does not begin for the regular 
students in the system until the third grade, therefore it would not be possible to have 
comparable historical data on elementary students until they reached the fifth grade. 

The hypothetical decision framework . In terms of the hypothesized decision making 
framework, the middle school principal's use of data very closely conformed to the model. 
He made, or encouraged, short term decisions based on classroom data and SBTR data. 
He used multiple points in time on sub-goals to work with parents to elicit their support in 
working with their children on specific weaknesses. His use of the BRAG system, cross 
referencing to a commercial instructional system, served as a perfect example of short term 
planning and decision making based on evaluative data. 

He also demonstrated the application of data in mid-term decision making through his 
use of PALT/SBTR data to make placement and forecasting decisions about students and 
to change those decisions based on further reviews of data. 

Long term decisions included deciding on the number of sections of algebra to offer 
the following year, how many Chapter 1 classes to plan, etc. For these, the middle school 
principal relied heavily on spring PALT data, broken out by goal area. 

The Teachers' Uses of Data 

Interviews and observations were conducted with four teachers from each of the two 
schools- the test coordinator and three teachers. Initial interviews were loosely structured 
and focused on types of decisions the teacher routinely made and the sources and types of 
data used in those decisions. A subsequent interview contained four types of questions: 
context, input, process and product. Context questions dealt with the types of decisions 
made, the teachers' feelings about their own competence in using data, and the level of 
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support received from the principal for data use. Input questions focused on how 
achievement data were used in decision making and the extent to which new decisions were 
being made based on data. For the middle school, additional questions were included on 
the use of SBTR and CAT. Process questions sought information on who participated in 
the decision making at various levels, extent and level of involvement of others in decision 
making, problems related to data use, and steps taken to resolve them. Product questions 
looked at inservice, services from the Evaluation Department, support needs, training needs 
and suggestions for improving PALT reports. 

The Elementary Teachers 

Two male and two female elementary teachers were interviewed and observed during 
the case study. Teachers were selected to obtain a balance across grade levels and by sex. 
One male and one female were third grade teachers; one male and one female taught fifth 
grade. Third grade is the first year that students are tested using the PALT. 

In general, the elementary teachers evidenced familiarity with a broad range of data and 
its uses. They were quite facile in their descriptions of the data they used and the decision 
making situations in which they used it. Very little prompting was needed to elicit detailed 
responses. The types of assessment data identified by them as being used rather routinely 
and the decisions supported by the data are shown in Table 8 on the following page. 

Types of Data Used 

The types of assessment data displayed in Table 8 are arranged in roughly ascending 
order from informal to formal, with the exception of the last category, enrollment data, 
which is a separate class of data entirely. 

It may be worth noting that none of the teachers identified, in the interview process, 
student self ratings or peer ratings as sources of data for decision making, although in one 
case a teacher did mention having students exchange papers to check their work. The 
distinction here is that the students were not rating one another's performance, but were 
assisting the teacher by checking papers. This practice also has obvious instructional 
benefits. None of these teachers used the phrase "performance assessment," even though 
they employed various forms of performance assessment, including both paper and pencil 
and observational techniques. In addition, only one of the four regular teachers indicated 
use of other teachers' opinions of students in making instructional decisions. 

Summary of Elementary School Teachers' Uses of Data 

The types of decisions routinely made by teacher.., not surprisingly, tended to focus on 
the instructional needs of students individually and in groups. These are illustrated by the 
following examples from the interviews and observations. The examples of reports that 
teachers referred to in describing their uses of data emphasized those that contained 
individual student data, especially those that included goal analysis data, such as the Class 
Alpha/Goal Report, which provides goal data by individual student within a class. 
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Table 8. 

Types of Assessment Data Used by Elementary Teachers for Decision Making 



Type of Assessment Data 

1. Teacher observation and judgment 

2. Oral question and answer 

3. Word games, e.g., "Who am I?" 



4. Assignments, homework, e.g. reading 
assignments, long range reports 

5. Unit tests (weekly) 

a. Textbook tests 

(MacMillan English, 
Heath Math) 

b. Student writing exercises; essays 



c. Student portfolios/work samples, 
e.g., spelling words in sentences 

6. PALT test data 



7. Enrollment data by class or program, 
e.g.. Chapter 1 



Uses 

1. How and what to teach 

2. Assess the student's understanding 
of material, expressiveness, ability 
to reason, and prove point 

3. Assess student's knowledge 
of material (memory); mostly 
in groups or individual. 

4. Assess knowledge/skill levels 



5. Evaluate instruction 

a. Screen to learn groups skill level; 
feedback to students on areas 
needing practice 

b. Assess subject knowledge and 
written expression 

c. Feedback to students on what 
they need to study 

6. Assess the strengths and weakness 
of the program 

a. Goal analysis 

b. Identify low achievers (Chapter 1) 

c. Placement in curriculum 

d. Evaluate growth 

7. Assign and schedule aides 
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Test-taking skills . One of the elementary teachers interviewed was a school leader in 
the area of test-taking skills. She simulated the PALT testing process in her daily 
instructional routine as illustrated in the following excerpt. 

/ do not teach the test, but I use the blue book (Parent-Teacher Guide) to help 
me ask questions like they do on the PALT tests. I'll phrase questions like 
they phrase them on the test and teach kids the instructions and give them the 
practice tests in the same format as the PALT 

rd like the test office to develop more practice tests in individual goal areas, 
so I could give students practice tests in just computation or word meaning. 

This same teacher also conducted a staff inservice workshop on test-taking skills at the 
elementary school. 

Test-taking preparation was a routine of this elementary school. At a staff meeting in 
mid-April, just before spring testing, the test coordinator handed out practice tests, 
encouraging teachers to use two practice tests before the districtwide testing was conducted. 
Teachers were also encouraged to read the directions for PALT Administration carefully 
and to plan their testing time very carefully. Instructions were given on testing absentees 
and on how to review answer sheets to remove stray marks and check for validity. That 
test-taking skills were imbedded in the curriculum was easy to observe in the classroom. 
Three of the observations witnessed instruction on or review of test-taking skills. 

Team planning . Team planning of instruction was the norm for this elementary school. 
Unfortunately, scheduling problems prevented us from observing the team planning process, 
but the teachers described it for us and made frequent references to the process in our 
interactions with them. Teachers plan in grade level teams. Tearn planning begins in the 
fall before school begins and planning sessions are conducted periodically throughout the 
year. In the initial fall sessions, individual student scores are examined in terms of their 
level by goal (Low, Average, High) in each of the three subject areas tested. Students with 
"L" scores are reviewed more closely for specific goal weaknesses. Instructional groups are 
organized so that areas of weakness can be addressed. 

End of year data use . Spring planning in the elementary school was aimed at making 
up class lists for the following fall. Tea ' srs reviewed PALT reports in combination with 
other data collected over the course to assign students to classes. Data on level of 
performance (L, AV, H) was combined with teacher observations regarding the need for 
supervision, independence, special program participation (Chapter 1, LD, TAG) to establish 
balanced classes. 

Within a class, a teacher may review the average RIT gain of his/her students and 
conduct a goal analysis to assess class and individual strengths and weaknesses. These goal 
analyses can be used for developing remediation efforts before the end of the year or for 
remediation of individual students in the fall. They can also be used to evaluate whether 
the average growth for the class is comparable to the school and the district, in much the 
same fashion as one would use norm-referenced data. 
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In using the goal analyses for remediation at year end, teachers typically combine PALT 
test results with textbook tests and professional judgment to diagnose specific needs. The 
same analyses can result in the enhancement of instruction for higher achieving students. 

Another device used by teachers at year end was a summary sheet that compared 
individual student RIT gains in each basic skills subject against the grade level average for 
the district and indicated whether the student was above or below the average. The 
principal prepared the comparison sheets and handed them out at a staff meeting the week 
before spring testing. Teachers were expected to finish filling out the forms from the 
reports they received back after spring testing. 

A Chapter 1 needs assessment was conducted in the spring to provide data for planning 
the fall program. The needs assessment involved two surveys which were administered by 
the school's Test Coordinator, who also served as the Chapter 1 coordinator. One survey, 
disseminated to staff and parents, asked for a rating of the need for each skill in a list of 
skills (no need, some need, great need) and other ancillary areas. A second survey, for 
teachers only, asked for a listing of student needs in reading and mathematics. 

Ability grouping . Grouping by ability level for math was a commonly observed practice 
in this school. Ijower level performance groups have alternate text materials and pullout 
groups were composed from across classes. This sometimes led to considerable disruption 
as students shuttled from room to room for special class sessions. Chapter 1 was conducted 
as a pullout program and pulling students out for other classes such as a computer lab was 
common practice at fifth grade. Reading instruction was conducted on a continuous 
progress basis. Data on student performance by goal and sub-goal was therefore used in 
placing students in math groups and monitoring their progress, but was not used in the same 
manner in reading. 

Classroom assessment . Classroom observations revealed a reliance on worksheets and 
the use of oral questioning strategies to assess student performance. Worksheets were used 
heavily by one of the third grade and one of the fifth grade teachers. They were also 
discussed and shared widely in staff meetings. It appeared that management and control 
of students was a problem in this school and that worksheets provided a means of keeping 
students on task. They were also used as enrichment exercises when students finished their 
in-class assignments within the allotted time. Oral questioning strategies observed in the 
fifth grade were mostly recall of knowledge. Less frequently, they required comparision of 
details and inferential responses from students, although some questions required 
formulation of opinions. Tlie fifth grade class was observed during a period when an 
English unit focused on the distinctions between opinion and fact; differences between truth 
and fiction were being taught. Observations of in-class social studies exercises also dealt 
with formulation of opinions, supported by fact. 

Other classroom assessment strategies used by the elementary teachers that were 
directly observed are listed below. 

1. Group reading. Teacher picks words or phrases from context and questions 
students about their meaning. 
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2. Demonstration/practice/reteach. The teacher demonstrated a skill and followed 
with observation and assistance, as needed; reteaching followed, based on the 
observation. Example: Italic handwriting was demonstrated on the board, followed 
by the teacher observing students' attempt to replicate the word. Individual 
assistance was given and a decision made to move on to another concept or reteach. 

3. Work sampling. Teacher reviewed samples of student work, such as spelling lists 
sampled for specific types of errors, such as capitalization, double consonants, etc. 

4. List of books read. Teacher used book lists as a means to check whether a student 
was progressing from less difficult to more difficult books. 

5. Writing journals. Journals were used as means of developing writing skills and 
providing the teacher with a record of progress shared with the student. Pages are 
kept, but not corrected. Journals are bound and given to students at end of year. 
Teacher can see how student has progressed in spelling, sentence construction, etc. 

6. Oral questions. The level of cognitive demand upon students varied according to 
the ability level of the group. 

The most common in-class instructional decision observed was to decide who to help 
and who to discipline. Data consisted of observations of immediate behavior, responses 
to teacher oral questions, or the questions asked by the student. 

The Middle School Teachers 

As indicated in the methods section, scheduling difficulties restricted us from obtaining 
the same level of in-depth data collection with the middle school as we were able to obtain 
from the elementary level. We conducted interviews with four middle school teachers, 
including the Test Coordinator, observed four classrooms, observed two sixth grade team 
meetings, and two staff meetings. 

Three female and one male middle school teachers agreed to participate in the case 
study. Teachers were selected to obtain a balance across grade level, subject area, and sex. 
Two females taught grade six; one taught math and one taught language arts. One male 
and one female taught grade eight; one taught math and language arts. 

In general the middle school teachers were more familiar with PALT test results and 
classroom assessment data than the elementary teachers. They were quite knowledgeable 
and sophisticated in their descriptions of the data and the instructional planning or decision 
making situations in which they used it. 

Types of Data Used 

The middle school teachers used an even broader range of assessment data than the 
elementary teachers. In addition to the types of data listed above for the elementary 
teachers, these teachers made use of group assessment methods, opinions of other teachers, 

29 



student self ratings and peer ratings. The pattern of use closely parallelled that of the 
middle school principal, with a somewhat greater emphasis on classroom assessment as 
would be expected. Specific data and examples of its uses are described below. 

Middle bchool teachers used PALT data at the beginning of the year to review student 
performance histories for patterns and consistency. The School-Based Test Reporting 
program provided teachers with two years of test data to review for patterns. One teacher 
assigned students many writing tasks early in the year and used the results to verify PALT 
performance and identify inconsistencies that could provide information for instructional 
planning. 

The language arts teachers gave few textbook or teacher-made tests because they didn't 
feel tests adequately reflected their model of breaking down instructional goals into very 
specific sub-components for the students. They did develop their own vocabulary tests to 
support the reading series. 

Teacher observation and judgments in the form of performance assessments were used 
quite effectively by one teacher in the following example. 

...My eighth grade students were preparing to write a research paper. I asked 
them, 'Wliat is knowledge? How do people get knowledge? Wliy do we need 
knowledge?'' The students listed places, people, and things that help them to 
acquire knowledge, then checked those they used most frequently, and those 
that were most accurate. This exercise on how to get knowledge most 
accurately led the students naturally into their research paper. I observe the 
process and student responses during discussion and get information on 
individual students and how they will approach the task 

Oral questioning was also used very effectively by this teacher who was a TESA 
(Teacher Expectations and Student Achievement) trainer. She was knowledgeable about 
how to ask questions to tap highei level cognitive processes. She used different questioning 
strategies for different students and topics, but asked all students opinion level (evaluative) 
questions. 

Assignments in this school tended to be those that could be done in-clabs. Very little 
homework was required because of the mixed ability levels in a given class. Essay 
assignments were evaluated primarily on content rather than style, spelling, or grammar. 
One teacher awarded A's to all assignments finished during the allotted class time. 

As an example of peer ratings, one teacher read essays aloud in class and the students 
voted to determine which one was best. This was done in conjunction with a writing class 
where students were taughi how to give a critical opinion of writing to peers. 

The most common classroom instructional decisions were deciding who to help and who 
to discipline. Teachers used data from behavioral observations, responses to teacher oral 
questions and the type of questions asked by students. 
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Summary of Middle School Teachers' Uses of Data 

The types and patterns of decision making by middle school teachers also followed the 
hypothesized decision model. PALTdata were used somewhat less in the fall for assigning 
students to classes and more for verifying or validating classroom assessments, than was the 
case in the elementary school. Classes were planned in the spring, based on four point-in- 
time data. The Principal's Final Goal Report from the fall testing provided data which, in 
combination with data from the BRAG, CRT and Portland Prognostic, could be used for 
re-assignment of students or for zeroing in on their specific learning needs. 

Over the course of the year, short range decisions at classroom level were supported by 
data from classroom assessments. Computerized Adaptive Testing (CAT), criterion 
referenced tests, and Basal Reading Assessment Guide (BRAG). 

Team planning . The middle school teachers met regularly in grade level or departmental 
planning teams. There appear to be three different types of planning. First, the "one- 
legged conference" consisting of informal contacts among teachers for sharing instruction 
on a particular topic. Second, formally structured team planning with designated time to 
plan, develop common topics, brainstorm ideas, and define resources. Third, specialized 
instructional units were divided among team members so each teacher planned one unit and 
taught it to all students in the grade level. This last strategy was being discussed by one 
eighth grade team. 

Also related to planning was the research teams formed by teachers within their 
departments or grade level teams to share "what works" for middle school instruction of 
minority learners. They addressed questions such as enhancing achievement by 
individualizing instruction, developing cooperative learning groups, and improving self- 
concept. The teachers focused on four effective strategies for improving middle school 
instruction of minority students in their school: 1) using team learning, 2) using student 
writing logs, 3) providing quality, targeted, motivational instruction, and 4) empowering 
student control of learning. 

Classroom assessment . Observations of classrooms indicated loosely structured, warm 
learning environments. Teachers relied on observation/judgment, oral questionning, 
teacher-made tests, and peer and self assessment ratings. Because classroom management 
impacts instructional decisions, lots of structure was emphasized at the beginning of the 
year. After the group dynamics are established, the looser structure with less reliance on 
textbooks, lots of use of overhead projector and teacher questions, and teacher-made tests 
come into use. There appeared to be little direct instruction, but an emphasis on one-to- 
one instruction. At the end of the year, again for management reasons, teachers returned 
to lots of structure. 

Other types of classroom assessment strategies routinely used by the middle school 
teachers and the decisions effected by the data included the following list. 

0 Teacher observation and judgment Who and what to teach; identifying learning 

styles and how student uses time 
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0 Classroom oral questionning 

0 Performance/portfolio assessment 

0 Group assessment methods 

0 Student self ratings/peer ratings 



Assessir g knowledge/understanding of 
content, providing feedback on student 
opinions, evaluating thinking skills. 

Reporting on students' goal progress 

Motivating group for cooperative learm'ng; 
group assessments were used for mid-term 
instruction (Nov-Apr); individual 
assessments used early fall and late spring 

Developing peer editing, clarifying writing 
criteria; providing critical opinion 



0 PALT test results Assessing goal strengths and weaknesses; 

grouping for instructional balance; 
placement in special programs; tracking 
student progress on graduation standards 

One teacher mentioned the need to provide assessment closure for students. She * 
provided a daily summary of the assessment activity linked to instruction in the class. 
Students were told of the assessment at the start of a unit, reminded throughout the process, 
and summaiized at the end. Strategics for this type of classroom assessment environment 
included student-to-student interaction, team tasks, simulation and games, independent 
study, and peer coaching. • 



DISCUSSION 

The Hypothetical Decision Framework 

The hypothetical decision framework posited at the beginning of this three year 
investigation breaks instructional decision making and planning activities into three levels 
based on the time horizon associated with each: short-term, mid-term and long-term. 
Decisions are then identified in terms of the decision maker and the level of the 
organization at which they are made: classroom decisions by teachers/aides; grade 
decisions by teachers/principals; school level decisions by principal; cluster level decisions 
by Director of Instruction; and district level decisions by central administration. 

The case studies focused on decision making and data use at the building and classroom • 
levels. We hoped to learn what decisions were made and what data were used in the 
process. The decision framework has proven to be a useful conceptual tool for viewing 
the decision - data use process. The findings of the case studies added to the validity of the 
framework and helped us to expand and refine the taxonomy. Further studies are needed 
to refine the framework if it is to serve as the basis for a model of data based decision 
making in a school district setting. 
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The Principals 



Previous survey results from the first study (Hansen, 1988; Mitchell & Hansen, 1987), 
revealed ('ifferences in the way principals and teachers used data that were consistent with 
their roles. The case study results were consistent with that earlier finding. This was 
especially evident in the survey data which showed clear differences in the types of metrics 
and reports preferred by principals and teachers. This finding was further supported by the 
interview and observation results which showed that the principals in this study, while 
varying in their involvement in the details of data interpretation, tended to focus on the 
whole school and to model good data usage for their teachers. The> not only set the tone 
for data use in their school, as one elementary teacher noted, but provided specific 
examples of appropriate usage and introduced tools for teachers to use in applying data to 
their instructional planning and decision making. 

Both principals were more dependent on the results of formal, standardized testing than 
were their teachers. This finding is consistent with the role expectations of teachers and 
principals and with their decision making horizons and focal areas as depicted in the 
hypothetical decision making framework. 

Principal supportiveness for teachers was perceived as high by teachers in both the 
elementary and middle school case studies. Teachers in both settings cited examples of how 
their principals used data to make decisions and how they proWded the teachers with useful 
analyses and summaries. The demonstrations of data use seemed to motivate teachers and 
elicit their respect, not only for the principal, but for the use of data as well. 

Much has been written about the changing role of the principal in vexeni years, most of 
which has focused on the role of the principal as instructional leader. Fewer observations 
have been made on the role of the principal in the information age. The accessibility/ of 
high speed, high capacity computers may be exerting a new influence on the role of the 
principal as a leader in the use of information to help teachers improve instruction. 
Portland Public Schools is one of a relative handful of schools using the type of technology 
exemplified by CAT and SBTR. As these types of technological innovations become more 
commonly available, teachers will have access to virtually continuous streams of electronic 
feedback on their students. They will need leadership that understands how to make the 
best use of this technology for planning and making decisions to help students learn. It is 
worth noting that before the case study was completed, the elementary principal notified the 
researchers that he was very interested in the CAT and SBTR systems and hoped that he 
could be one of the first elementary schools to be involved with these systems. 

The Teachers 

Teachers in the study were mostly skilled data users, just as they had identified 
themselves to be in the survey. Some differences were evident in the types of data available 
to teachers at the two levels, and their uses followed from the differences. The availability 
of CAT (at a neighboring school) and SBTR at the middle school, naturally resulted in 
more reliance on PALT data for middle school teachers. As suggested earlier, the teachers 
at both schools tended to follow the lead of their principal when it came to data use. 
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In the survey, teachers expressed a strong preference for classroom assessment data. 
This is consistent with findings by Stiggins (1985). Interviews and observations, especially 
at the elementary level, revealed that classroom practices were consistent with the survey 
findings. Stiggins and others who have intensively studied the classroom assessment 
environment have found standardized tests to be among the least useful forms of 
measurement for instructional decisions. However, i ^^'^uld be clear that the Portland 
Achievement Levels Test (PALT) is not just anoth^ '^'^ lardized test. It is specifically 
designed to measure the breadth anc' :cope of the k curriculum and is not in any 

way dependent on a norming population for interpretation. The PALT has been 
identified by Haney (1985) as one of the most educationally useful measurement 
programs in a school district anywhere in the country. The Computerized Adaptive 
Testing system utilizes items from the PALT and produces scores which are valid, low 
error estimates of performance on the PALT. This results in an instructionally sensitive 
feedback system that can provide individual teachers with a quick and accurate 
assessment of student mastery of curriculum objectives. 

The survey and case study results were consistent regarding the types of reports 
used by teachers and confirmatory of their preference for data on individual students. 
Teachers' preference for reports was a function of the level of detail the report provided 
on individual students. Their most frequently used reports were those that provided data 
on individual student goal attainment. The more detailed and specific the analysis of 
data, the more useful the teachers perceived the report to be. This is not surprising in 
the least, but tends to confirm that good teachers focus more on the needs of individual 
students than on those of a group. 

The findings regarding types of reports preferred by school level varied in ways that 
are difficult to explain. Elementary teachers reported greater use of fall data from the 
principals' reports and saw less value in the Parent-Teacher Conference report, and 
labels for cum-folders. They saw more value in the Grade Goal Progress Report and 
Five Year Achievement Gains. Middle school teachers were less dependent on fall data 
and preferred the Class Alpha/Goal Report and Principal's Alpha. These differences 
could be attributable, at least in part, to differences in the principals' preferences. 
Principal style, interest in data, facility with data interpretation and use, and leadership 
certainly seemed to play a significant role in determining the patterns of use among the 
teachers. 



SYNTHESIS AND RECOMMENDATIONS 

Several impressions and interpretations emerge from this study that are not so much 
a product of the data as they are borne of the researchers' experience in conducting the 
case study. They are listed below in no particular order of importance. 

1) The;^ are principals and teachers who are effective data users, contrary to popular 
myth. These educators understand the value of the data available to them and make 
good use of it in their daily routines. We found more evidence of data use than 
anticipated, even among the high data use case study schools. 
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2) A key factor in whether a school is a high data use school or not is the principal's 
attitude toward and comfort level with data. Principals set the tone and create the ethos 
for data use by the way they model the use of data in the planning and decision making 
process. We did not explore the issue of why the case study principals were high data 
users or how they developed into data users. 

3) Questions remain whether the behaviors of the principals and teachers in these high 
data use schools are observably different from those in lower data use schools. How 
different are they? In what ways do they differ? The answer to these questions would 
help us to understand how data users develop and how the central office support system 
might better address their needs. Further study is needed to explore these questions. 

4) High student transiency affected, possibly in a restrictive fashion, the flexibility of 
data use among teachers in the case study schools. One teacher noted the complete 
turnover of his lower level math group of 15 students between September and February. 
The effects of this type of transiency on data use are not well known. One can 
speculate that the decision framework in such cases would be more restricted to the 
short-term, placing a greater emphasis on classroom and criterion referenced forms of 
assessment. 

5) The Research and Evaluation Department rhould consider developing a unified 
report of Portland Achievement Levels Tests score results, science testing, and direct 
writing assessments. 

6) The Research and Evaluation Department should develop an Assessment Guide for 
teachers and principals which might include information on (a) the role of testing and 
assessment, (b) development of effective classroom assessments, (c) preparation of 
students for testing, (d) communication with parents about assessment, and (e) use and 
interpretation of Portland Achievement Levels Tests (PALT). Training and dissemination 
of the Assessment Guide should also be implemented by the Department. 

7) The Research and Evaluation Department should revise the Test Coordinator's 
Manual. The manual might include information on (a) philosophy of testing in Portland 
Public Schools, (b) interpretation and use of PALT test results, (c) descriptions of the 
PALT testing program, (d) logistics of testing, (e) facsimiles of PALT test reports, and 
other information applicable to Test Coordinators. 

During 1989, the results of the Decision Making study are being used to design 
training and support materials for effective use of data in schools. We hope to continue 
to explore the utility of the decision making framework posited for this study in other 
settings. This may necessitate revising the taxonomy to more closely correspond to the 
types of data available in a specific district or school. 

The application of appropriate, timely, high quality information to the instructional 
decision making process suggests a circle that begins and ends with student learning. All 
too often this circle has been interrupted at the principal-teacher level due to 
inappropriate information or misapplication of appropriate data to the improvement of 
instruction. The promise of this research study is that it offers a realistic means of 
assuring that quality data are appropriately applied in completing that circle. 
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Endnotes 



1. RIT stands for Rasch unlT. A unit of measurement named for the Danish 
mathematician Georg Rasch, who developed the single parameter model of scaling test 
items, based on their log-item easiness. The RIT scale is a continuous, equal interval 
scale tied to the Portland curriculum. It ranges from approximately 140 to 280, spanning 
grades three through eight. 

2. P-score stands for Portland Score, a standard score with a mean of 50 and a standard 
deviation of 10. It is developed separately for each grade and subject encompassed by 
the PALT each time the tests are given. 

3. The Grade Goal Progress Report for a school shows the fall and spring data for each 
curriculum goal by grade in RIT means and their deviations from the districtwide mean 
for that goal and grade, as well as the fall-to-spring growth for the school and its relation 
to the district average growth. Deviations of .5 or more are considered meaningful. 
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USE OF EVALUATIVE DATA IN INSTRUCTIONAL DECISIONMAKING 



CASE STUDY SURVEY - February 1988 



Please circle the extent to which you use these data sources 
in instructional planning and decisionmaking. 



PALT tests 

Other standardized tests 
Teacher-made tests 

Textbook tests/District adoption inventories 

Performance assessments 

Oral questioning strategies 

Group assessment methods 

Opinions of other teachers 

Assignments 

Student peer rating 

Student self rating 

Markham Math Diagnostic Test 

BRAG: Basal Reading Assessment Guide 

Computerized Adaptive Testing (CAT) 

School Based Test Reporting (SBTR) 

Other: 



NEVER SOME ALWAYS 
SELEXDM OFTEN 



1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 



2 
2 
2 
2 
2 
2 
2 
2 

2 
2 
2 
2 
2 
2 
2 
2 



3 
3 
3 
3 
3 
3 
3 
3 
3 
3 
3 
3 
3 
3 
3 
3 



4 
4 
4 
4 
4 
4 
4 
4 
4 
4 
4 
4 
4 
4 
4 
4 



5 
5 
5 
5 
5 
5 
5 
5 
5 
5 
5 
5 
5 
5 
5 
5 



Check the PALT reports you use. 

Circle Lhe extent of use of the reports that you checked. 



Student Achievement Reports: 

Student Goal Report by class (alpha) 

Test Score Cards 

Parent-Teacher Conference Report 

Preliminary Principal's Report (alpha) 

Preliminary Principal's Report (rank) 

Final Principal's Report - Fall 

Fall-Spring Match Report by grade 

Labels for Student Cum Folders 

Middle School Alpha Feeder Report 

Middle School Rank Feeder Report 

Administrative Reports: 

Goal Progress Report by grade 

Report on Five Year Achievement Gains 

Report on Board Achievement Goal #1 

Report on Board Achievement Goal #2 

Achievement Report by Ethnic Group 

Five Year Growth Charts 

Achievement Profiles 



1 
1 

1 
1 
1 
1 
1 
1 
1 
1 



1 
1 
1 
1 
1 
1 
1 



2 
2 
2 
2 
2 
2 
2 
2 
2 
2 



2 
2 
2 
2 
2 
2 
2 



3 
3 
3 
3 
3 
3 
3 
3 
3 
3 



3 
3 
3 
3 
3 
3 
3 



4 
4 
4 
4 
4 
4 
4 
4 
4 
4 



4 
4 
4 
4 
4 
4 
4 



5 
5 
5 
5 
5 
5 
5 
5 
5 
5 



5 
5 
5 
5 
5 
5 
5 



To what extent do you use evaluative data in 
instructional planning and decisionmaking? 



NEVER SOME ALWAYS 
SELDOM OFTEN 

4. To what extent do you use evaluative data to 

develop classroom or building academic goals? 12 3 4 5 

5. To what extent does sharing assessment data 

with students positively effect achievement? 12 3 4 5 

6. Check the decisions in which you use evaluative data in your role. 
Circle the type of test score you use in the situations that 

you checked (RIT score, P-score, Other score, or Not Applicable) . 



Goal setting by class or building 


RIT 


P 


0 


NA 


Student placement/scheduling 


RIT 


P 


0 


NA 


Grouping/regrouping students 


RIT 


P 


0 


NA 


Curriculum evaluation 


RIT 


P 


0 


NA 


Public communication 


RIT 


P 


0 


NA 


Reviewing individual student growth 


RIT 


P 


0 


NA 


llocating funds 


RIT 


P 


0 


NA 


otudent promotion 


RIT 


P 


0 


NA 


Communicating with parents 


RIT 


P 


0 


NA 


Communicating with teachers 


RIT 


P 


0 


NA 


Reporting to the district 


RIT 


P 


0 


NA 



7. PALT test scores are grouped in different formats. 
Check the two (2) formats most useful to you. 



Individual student 

Instructional group 

Class 

Grade 

School 

Cluster 

District 



8. To what extent would you use expanded 

technical assistance or inservice of the 
Evaluation Department, if these services 
were available to you? 



9. Do the PALT reports you receive meet all of 

your instructional decisionmaking needs? Y N 

If no, what other types of information would 

you find useful? For what purposes? 
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10. Would you like to receive more information 

on other types of assessment besides PALT? Y N 



If yes, what types? 



11. If you had one wish or one change you could make in the 
PALT test reports you receive, what would you wish? 



12. Additional comments: 



PLEASE COMPLETE THE FOLLOWING INFORMATION: 



1. Type of 
school : 



K-5 
6-8 



4. Sex: 



Female 
Male 



2. Role: 



Principal 
Asst. Prin, 
Teacher 
Test Coord • 



5* Age: 



30-39 
40-49 
50-59 
60-69 



3- 



Years 
in 

Role: 



1-5 
6-12 

13-19 

2 0+ 



6. 



Highest 
Degree: 



Bachelor 

Master 

Doctorate 



Name (optional) 



PLEASE RETURN TO STEPHAl^IE MITCHELL, EVALUATIO^' DEFT, BESC 

THANK YOU. 



CASE STUDY SURVEY 
Question 1 



EVALUATIVE DATA SOURCES USED 
FOR INSTRUCTIONAL PLANNING 
(Rank Order) 


Never/ 
Seldom 


Some 


Always/ 
Often 


Performance Assessment 






6 


1 


93.9 


Oral Questioning Strategies 


6, 


3 


6 


3 


87.6 


Teacher-Made Tests 


8 


6 


17 


1 


74 .2 


Assignments 


18 


8 


15 


6 


65. 6 


Textbook Tests 


23 


5 


29 


4 


47.0 


Other 


54 


4 






45.5 


Opinions of other Teachers 


14 


7 


41 


.2 


44 . 1 


PALT Tests 


37 


2 


20 


.0 


42.9 


Group Assessment Strategies 


15 


2 


48 


.5 


36.4 


Other Standardized Tests 


37 


1 


28 


.6 


34 . 3 


Student Self Rating 


45 


.1 


25 


.8 


29.1 


Markham Math Diagnostic 


74 


.2 


3 


. 2 


22 . 6 


Student Peer Rating 


71 


.9 


15 


.6 


12.5 


School Based Test Reporting 


83 


.3 


10 


.0 


6.6 


BRAG: Basal Reading Assessment 


90 


.6 


3 


.1 


6.3 


Computerized Adaptive Testing 


96 


.6 






3.4 



CASE STUPY SURVEY 
Question 2 



PALT REPORTS USED FOR 
INSTRUCTIONAL PLANNING 


Never/ 
Seldom 


Some 


Always/ 
Often 


Achievement Reports (Rank Order) 


Fall -Spring Match Report by grade 


25 


.0 


25 


.0 


50 


0 


Student Goal Report/Class Alpha 


21 


*7 


30 


.4 


47 


.8 


Preliminary Principals - Rank 


35 


.0 


30 


.0 


35 


.0 


Parent-Teacher Conference Report 


44 


«4 


22 


.2 


33 


.3 


Preliminary Principals - Alpha 


34 


.7 


34 


.8 


30 


.4 


Final Principals Report - Fall 


40 


.0 


30 


.0 


30 


.0 


Test Score Cards 


50 


.0 


20 


.0 


30 


.0 


Labels for Student cum Folders 


42 


.9 


28 


.6 


28 


.6 


Middle Schocl Alpha Feeder 


76 


.9 






23 


.1 


Middle School Rank Feeder 


60 


.0 


20 


.0 


20 


.0 


Administrative Reports (Rank Order) 


Goal Progress Report by Grade 


33 


. ? 


19 


.0 


47 


.6 


Achievement Profiles 


61 


.1 


11 


.1 


27 


.8 


Five Year Achievement Gains Rpt. 


63 


.2 


15 


.8 


21 


.1 


Five Year Growth Charts 


62 


.6 


18 


.8 


18 


.8 


Achievement Rpt. by Ethnic Group 


81 


.3 


6 


.3 


12 


.5 


Board Rpt. Achievement Goal #1 


86 


.7 


6 


.7 


6 


.7 


Board Rpt* Achievement Goal f2 


86 


• 7 


6 


.7 


6 


.7 



DO 



CASE STUDY SURVEY 
Question 3 



USES OF DATA FOR 
INSTRUCTIONAL PLANNING 


Never/ 
Seldom 


Some 


Always/ 
Otten 


Planning or Making Decisions 


14.3 


28 . 6 


57.1 


Developing Class/Building Goals 


17.6 


26.5 


55 . 8 


Sharing Data with Students 
to Improve Achievement 


14.3 


31 4 


54. 3 



CASE STUDY SURVEY 
Question 4 



DECISION SITUATIONS 


RIT 


P 




RIT 
& P 


RIT & P & 
Other Other 


All 


Other 


N/A 




Setting class/Building Goals 


17.4 


8. 


7 


17.4 


4.3 4.3 


4 . 3 


4*3 


39. 


1 


Student Placement/Scheduling 


16.0 


24, 


0 


24 .0 




12 .0 


4.0 


20. 


0 


Grouping Students/Regrouping 


14.8 


18, 


5 


11. 1 


3.7 


14 .8 


14 . 8 


22. 


2 


Curriculum Evaluation 


12.5 


25, 


0 


12.5 




12.5 




37. 


5 


Public Communication 








31.3 


6.3 


6.3 


12.5 


43. 


8 


Reviewing Student Growth 


21.7 


13 


0 


13.0 




17.4 


13.0 


21. 


7 


Allocationg Funds 




11 


1 










88. 


9 


Student Promotion 


12.5 


6 


.3 


25.0 






6.3 


50. 


0 


Communicating w/Parents 


7.1 


14 


.3 


39. 3 




10.7 


10.7 


17. 


9 


Communicating w/Teachers 


8.7 


13 


.0 


39. 1 




8.7 


8.7 


21. 


7 


Reporting to the District 




12 


.5 


25.0 


6.3 


6.3 


6.3 


43. 


8 



51 



CASE STUDY SURVEY 
Question 5 



TEST REPORT FORMAT 
(Rank Order) 


Preferred 
Format 


Individual Student 


63.9 


Instructional Group 


36.1 


Class 


27.8 


Grade 


22.2 


School 


2.8 


Cluster 




District 





CASE STUDY SURVEY 
QUESTION 6 



INTEREST/USE OF EXPANDED 
EVALUATION TECHNICAL ASSISTANCE 


NEVER/ 
SELDOM 


SOME 


ALWAYS/ 
OFTEN 


• 


ALL SITES 


30.0 


33.3 


36.7 












• 


CASE STUDY SURVEY 
QUESTION 7 






• 


DO PALT REPORTS MEET YOUR 
DECISION MAKING NEEDS? 


YES 




NO 




ALL SITES 


42.3 




57 .7 
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CASE STUDY SURVEY 

Survey Question 1 









ELEMENTARY 


SCIfOOL 






MIDDLE 


SCHOOL 


EVALUATIVE DATA USED FOR 


Never/ 


Some- 


Always/ 


Never/ 


Some- 




Always/ 




Seldom 


times 


Often 


Seldom 


times 


Often 


cAijx xesus 


33 


. 4 


8 


. 3 


58 


.4 


1 39 


.1 


26 


.1 


34 . 7 


KJl^ilGL i> v-anQajTulzeu Tests 


41 


. 6 


25 


.0 


33 


. 3 


34 


.7 


30 


.4 


34 . 8 


xeaufiex^rlaKe TGSuS 


16 


.7 


33 


. 3 


50 


.0 


4 


.3 


8 


.7 


87.0 


To Vt" Hr>r>lr To e e 
Xc^ Ui<^IJIJlV xesus 


27 


. 3 


9 


1 


63 


.6 


21 


.7 


39 


.1 


39. 1 


Per f ozmance Assessment 






9 


1 


91 


.0 






4 


.5 


95.4 


\jLaj. yutf b V.X onin(j 5uruue(jics 










100 


.0 


8 


.7 


8 


7 


82.6 


v^wu^ /Assess III ciiu ouxaueuxes 


9 


, 1 


54 , 


5 


3 6 


^ 4 


1 Q 
XO 








36.4 


upxnxoDb ox otner leacners 


18 


2 


36. 


4 


45 


5 


13 


0 


43 


5 


43. 4 




20 


0 


30. 


0 


50 


0 


18 


.1 


9 


1 


72.7 


St"n rfpn t" Do o r" Pat" \ r»rf 


72 


8 


18. 


2 


9 


1 


71 


4 


14, 


3 


14.3 


Student Self Rating 


54 


6 


27. 


3 


18 


2 


40 


0 


25, 


0 


35 . 0 


Markham Math Diagnostic 


100, 


0 










61, 


9 


4. 


8 


33.3 


BRAG: Basal Reading Assessment 


90. 


9 


9. 


1 






90, 


5 






9.5 


Computerized Adaptive Testing 


100. 


0 










94 , 


7 






5.3 


School Based Test Reporting 


100. 


0 










75. 


0 


15. 


0 


10.0 


Other 


50. 


0 






50. 


0 


57. 


1 






42.9 
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CASE STUDY SURVEY 

Survey Question 2 



USED ^gg^^^EMENT REPORTS 
INSTRUCTIONAL PLANNING 



ELEMENTARY SCHOOL 



Never/ 
Seldom 



Student Goal Report/Class Alpha 
Test Score Cards 

Parent-Teacher Conference Report 
Preliminary Principals-Alpha 
Preliminary Principals-Rank 
Final Principals Report-Fall 
Fall-Spring Match Report by Grade 
Labels for Student Cum Folders 
Middle School Alpha Feeder 
Middle School Rank Feeder 



Some- 
times 



Always/ 
Often 



USED ^g^^^"^STRATIVE REPORTS 
INSTRUCTIONAL PLANNING 



40.0 
66. 6 
55.5 
30.0 
50.0 
37.5 
25.0 
14 . 3 
100.0 
100.0 



Never/ 
Seldom 



10.0 
11.1 
11.1 
30.0 

12.5 
12.5 
42.9 



50.0 
22.2 
33.3 
40.0 
50.0 
50.0 
62.5 
4r . 0 



MIDDLE SCIfOOL 



Never/ 
Seldom 



Some- 
tinies 



7.7 

36.4 

38.9 

38.5 

25.0 

41.6 

25. 1 

57.1 

70.0 

50.0 



46.2 
27. 3 
27.8 
38.5 
50.0 
41.7 
31.3 
21.4 

25.0 



Some- 
times 



Goal Progress Report by Grade 22.2 22.2 

Five Year Achievement Gains Rpt. 50.0 1-^.5 

Board Rpt. Achievement Goal #1 80.0 20.0 

Board Rpt. Achievement Goal #2 80.0 20.0 

Achievement Rpt. by Ethnic Group 83.3 16.7 

Five Year Growth Charts 50.0 16.7 

Achievement Profiles 57.1 14^3 



Always/ 
Often 



55.5 
37.5 



33.3 
28.6 



Never/ 
Seldom 



Some- 
times 



16.7 
72.7 
90.0 
90.0 
80.0 
70.0 
63.7 



16.7 
18. 2 



20. 0 
9.1 



53 



Always/ 
Often 



46.1 
36.4 
33.3 
23.1 
25.0 
16.6 
43.8 
21.4 
30.0 
25.0 



Always/ 
Often 



41.7 
9.1 
10.0 
10.0 
20.0 
10.0 
27.3 



• 



CAGE STUDY SURVEY 

Survey Question 4 

















MIDDLE 


SCHOOL 


DECISIONS SITUATIONS 


RIT 


P 






RIT/ 
P 


RIT/ 
OTHER 


OTHER ALL 


Setting Goals by Class/Bldg. 


14.3 


14 


.3 


35 


.7 


14.3 


7.1 


7.1 


Student Placement/Scheduling 


6.7 


26 


.7 


20 


.0 


26.7 


6.7 


13.3 


Grouping/Regrouping students 


5.6 


10 


7 


22 


2 


16.7 


22.2 


5.6 11.1 


Curriculum Evalua :ion 




44 


4 


33 


3 


22.2 






For Public Cc.iu.iuniv.ation 








40, 


0 


40.0 


20.0 




For Reviewing student Growth 


21.4 


14, 


3 


21, 


4 


14.3 


14.3 


14.3 


For Allocating Funds 




20, 


0 


80, 


0 








For Student Promotion 


11.1 






44. 


4 


33.3 


11.1 




Communicating with Parent^i 


5.9 


11. 


8 


17. 


6 


41.2 


17. 6 


5.9 


Communicating with Teachers 


7.1 


14. 


3 


21. 


4 


42.9 


14.4 




Reporting To The District 




22. 


2 


33. 


3 


33.3 


11.1 





CASE STUDY SURVEY 

Survey Quest on 4 



DECISIONS SITUATIONS 



ELEMENTARY SCHOOL 



RIT 



Setting Goals by Class/Bldg. 22.2 

Student Placement/Scheduling 30.0 

Grouping/Regrouping Students 33.3 

Curriculum Evaluation 28.6 
For Public Communication 

For Reviewing Student Growth 22.2 
For Allocating Funds 

For student Promotion 14.3 

Communicating With Parents 9.1 

Communicating with Teachers 11.1 
Reporting To The District 



20.0 
22.2 



11.1 

14.3 
18.2 
11.1 



NA 



44.4 
20.0 
22.2 
42.9 
50.0 
22.2 
100.0 
57.1 
18.2 
22.2 
57.1 



RIT/ RIT/ 
P OTHER 



22.2 
20.0 



16.7 
11.1 

14.3 
36.4 
33.3 
14.3 



11.1 



16.7 
11.1 



OTH^R 



ALL 



14 .3 



10. 0 
22. 2 
28. 6 
16.7 
22. 2 



18.2 
22.2 
14.3 



5.; 



# 



CASE STUDY SURVEY 

Survey Question 3 







ELEMENTARY 


SCHOOL 


MIDDLE SCHOOL 


USED DATA FOR 
INSTRUCTIONAL PLANNING 


Never/ 
Seldom 


Some*- 
times 


Always/ 
Otten 


Never/ 
Seldom 


Some- 
times 


Always/ 
Often 


Data to Plan or Make Decisions 




18.2 


81.8 


20.9 


33.3 


45.9 


Data to Develop Class/Bldg Goals 




18.2 


81.8 


26.1 


30.4 


43. 5 


Share Data w/Students to Improve Achvmnt 


9.1 


27.3 


63.6 


16.7 


33.3 


50.0 



CASE STUDY SURVEY 

Survey Question 5 



• 








ELEMENTARY 


SCHOOL 


MIDDLE SCHOOL 










Yes 




No 


Yes 




No 




Individual Student - 


Most Useful Format 


100. 


0 




100. 


0 






Instructional Group 


- Most Useful Format 


100. 


0 




100. 


0 






Class - 


Most Useful 


Pa It Format 


100. 


0 




100. 


0 




Grade - 


Most Useful 


Pa It Format 


100. 


0 




100. 


0 






School - 


■ Most Useful 


Pa It Format 


100. 


0 












Cluster 


- Most Usefu 


1 Palt Format 
















Disrtict - Most Useful Pelt Format 















CASE STUDY SURVEY 

Survey Question 6 



ELEMl RY SCHOOL 


MIDDLE SCHOOL 


SOi Always/ 
Seldom timtiS Often 


Never/ Some- Always/ 
Seldom times Often 


Extent Use Expanded Tech. Assistant 11.1 66.7 22.2 


38.1 19.0 42.8 


CASE STUDY SURVEY 

Survey 7 


ELEMEIiTARY SCHOOL 


MIDDLE SCHOOL 


Yes No 


Yes No 


PALT Rpts. Meet Decisionmaking Needs 25.0 75.0 


50.0 50.0 
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CASE STUDY PROTOCOL 

Part I. Principal's Name School 

Thank you for agreeing to participate in a case study 
interview. As part of a district wide study of evaluation use, 
we gathering perceptions on how evaluation data are used in 
instructional decisions • The research is designed to guide 
policy decisions about how data are used at the building level. 

Our operational definition of evaluative data includes a 
wide range of data sources, such as PALT tests, other 
standardized tests , teacher-developed tests , textbook tests , 
teacher observation and judgement of student performance, 
assignments, grades, etc. All your responses will be treated 
confidentially. Do you have any questions before we begin? 

Part II • Administrator Interview Questions 

Context Questions 

1. What decisions, if any, do you routinely make using data? 

2. How do you feel about your competence in using data for 
instructional planning and decision making? 

3. How do your teachers feel about the use of evaluative 
information for instructional planning and decision making? 

Input Questions 

4. How do you use achievement data to make instructional plans 
or decisions for the building? 

5. Have any new decisions been made based on available data? 

6. Has the School Based Testing Program changed anything in how 
you make instructional decisions? 

7. Has the Computer Adaptive Testing (CAT) program changed 
anything in how you make instructional decisions? 

Process Questions 

8* Who participates in instructional planning and decision 
making at the building level? 

9. To what extent, are others involved in the decision making? 
What is the nature of their involvement? 

10. What are the potential or actual problems related to using 
data to plan instruction? 

11. What steps have been taken to resolve these problems? 

ERLC 



Product Questions 

12. What is the extent of inservice available to yon in using 
evaluative data to plan instruction or inform decisions? 

13. How can measurement and evaluation more effectively serve 
your decision making needs? 

14. What services, reports, and training are needed to make the 
most effective use of data in your decision making? 

15. What suggestions do you have for improving PALT reports? 

16. What are the positive and negative aspects of using data for 
instructional planning and decision making? 

17. What factors are associated with the success or failure of 
using evaluation data for instructional decision making? 

18. What kind of training or support would you find most helpful 
in improving the collection and use of data to make 
instructional decisions? 



Thank you for your participation. We appreciate your taking time 
from your busy schedule to assist with this study. 



CASE STUDY PROTOCOL 



Section I. Introductory Comments 

Teacher's Name School 

Thank you for agreeing to participate in a case study 
interview. As part of a district wide study of evaluation use 
we are interested in gathering your perceptions of how evalual-ioA 
data are used in instructional decisions. The research is 
designed to guide and inform policy decisions about how data are 
used at the classroom and building level. 

Our operational definition of evaluative data includes a 
wide range of data sources, such as PALT tests, other 
standardized tests, te ache r - deve 1 ope d tests, textbook tests, 
teacher observation and judgement of student performance 
assignments, grades, etc. All your responses will be treated 
confidentially. Do you have any questions before we begin? 

Section II. Instructional Staff Interview Questions 
Context Questions 

1. What decisions, if any. do you routinely make using data? 

2. How do you feel about your competence in using data for 
instructional planning and decision making? 

3. How does your principal support the use of evaluative 
information for instructional planning and decision making? 

Input Ques tions 



4 

5 
6 



How do you use achievement data to make instructional plans 
or decisions for your class? 

Have any new decisions been made based on available data? 

Has the School Based Testing Program changed anything in how 
you make instructional decisions? 

7. Has the Computer Adaptive Testing (CAT) program changed 
anything in how you make instructional decisions? 

Process Ques tions 

8. Who participates in instructional planning and decision 
making at the classroom level? 

9. To what extent, are others involved in the decision making? 
What is the nature of their involvement? 



10. What are the potential or actual problems related to using 
data to plan instruction? 

11. What steps, if any, have been taken to resolve the problems? 
Product Ques t ions 

12. What is the extent of inservice available to you in using 
evaluative data to plan instruction or inform decisions? 

13. How can measurement and evaluation moire effectively serve 
your decision making needs? 

14. What services, reports, and training are needed to make the 
most effective use of data in your decision making? 

15. What suggestions do you have for improving PALT reports? 

16. What are the positive and negative aspects of using data for 
instructional planning and decision making? 

17. What factors are associated with the success or failure of 
using data for instructional decision making? 

18. What kind of training or support would you find most helpful 
in improving your collection and use of data to make 
instructional decisions? 



Thank you for your participation. We appreciate your taking time 
from your busy schedule to assist with this study. 



